Sealbook / Open source / Databases & data engineering

gnemoug/distribute_crawler

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

License & maintenance

No license stated: by default nobody may reuse the code. Ask the authors before using it in a product.

Last code push 9 years ago.

Similar projects

	Repository	Stars	Language	License	Activity
1	supabase/supabaseThe Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.	★ 111.7k	TypeScript	Apache-2.0	Active
2	django/djangoThe Web framework for perfectionists with deadlines.	★ 91.4k	Python	BSD-3-Clause	Active
3	doocs/advanced-java😮 Core Interview Questions & Answers For Experienced Java(Backend) Developers \| 互联网 Java 工程师进阶知识完全扫盲：涵盖高并发、分布式、高可用、微服务、海量数据处理等领域知识	★ 79.2k	Java	CC-BY-SA-4.0	Active
4	elastic/elasticsearchFree and Open Source, Distributed, RESTful Search Engine	★ 78.3k	Java	Other	Active
5	obsproject/obs-studioOBS Studio - Free and open source software for live streaming and screen recording	★ 77.7k	C	GPL-2.0	Active
6	redis/redisFor developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data stru	★ 76.9k	C	Other	Active
7	pathwaycom/pathwayPython ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.	★ 62.1k	Python	Other	Active
8	meilisearch/meilisearchA lightning-fast search engine API bringing AI-powered hybrid search to your sites and applications.	★ 59.6k	Rust	Other	Active