SocketlistSearch & web / polite-crawler
polite-crawler
io.github.mark1russell7/polite-crawler · v0.1.0
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
Install
npx -y polite-crawlerTypical command from the package name. Check the README for required configuration and credentials before connecting it to real data.
Signals
| GitHub stars | 0 |
| npm weekly downloads | 0 |
| Last push | 2d ago |
| Licence | MIT |
| Open issues | 0 |
| SECURITY.md | Yes |
| Language | TypeScript |
| Latest version published | 2026-10-13 |
Signals captured from GitHub, npm and the official MCP Registry on October 15, 2026. See the scoring method.
Similar servers
fastCRW
Score 75Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
web-research-mcp
Score 74AI-powered web research MCP server for agents. Supports web search, browse page, extract...
Open Agent Search
Score 74Search the web, news, images, videos and books, and extract page content for AI agents.