adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

article-extractorcorpus-buildercorpus-toolscrawlerhtml-to-markdownhtml2textllmnews-aggregatornews-crawlernlp

License & maintenance

Apache-2.0 is a permissive license: free to use commercially with attribution.

Last code push 12 days ago.

	Repository	Stars	Language	License	Activity
1	firecrawl/firecrawlSupercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥	★ 192.7k	TypeScript	AGPL-3.0	Active
2	unclecode/crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Clo	★ 85.8k	Python	Apache-2.0	Active
3	google/langextractA Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visua	★ 39k	Python	Apache-2.0	Active
4	ScrapeGraphAI/Scrapegraph-aiPython scraper based on AI	★ 31.9k	Python	MIT	Active
5	AiHubCN/Awesome-Chinese-LLM整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。	★ 22.8k	—	—	Slowing
6	affaan-m/ECCThe agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Code	★ 283.1k	JavaScript	MIT	Active
7	Significant-Gravitas/AutoGPTAutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on w	★ 187.6k	Python	Other	Active
8	ollama/ollamaGet up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.	★ 183.7k	Go	MIT	Active