oxylabs/ai-crawler-py

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

aiai-agentsai-crawlerai-scrapingai-studioai-web-crawlercrawl-agentweb-crawler

License & maintenance

No license stated: by default nobody may reuse the code. Ask the authors before using it in a product.

Last code push 59 days ago. Archived: read-only, no further updates.

	Repository	Stars	Language	License	Activity
1	firecrawl/firecrawlSupercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥	★ 192.7k	TypeScript	AGPL-3.0	Active
2	NousResearch/hermes-agentThe agent that grows with you	★ 254.9k	Python	MIT	Active
3	langchain-ai/langchainThe agent engineering platform.	★ 148k	Python	MIT	Active
4	google-gemini/gemini-cliAn open-source AI agent that brings the power of Gemini directly into your terminal.	★ 107.4k	TypeScript	Apache-2.0	Active
5	thedotmack/claude-memPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and inject	★ 101.6k	TypeScript	Apache-2.0	Active
6	infiniflow/ragflowRAGFlow is a leading open-source Retrieval-Augmented Generation (RAG) engine that fuses cutting-edge RAG with Agent capabilities to create a	★ 92.4k	Go	Apache-2.0	Active
7	D4Vinci/Scrapling🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg	★ 88.4k	Python	BSD-3-Clause	Active
8	unclecode/crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Clo	★ 85.8k	Python	Apache-2.0	Active