istresearch/scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

distributedkafkapythonredisscrapingscrapy

License & maintenance

MIT is a permissive license: free to use commercially with attribution.

Last code push 2 years ago. Archived: read-only, no further updates.

	Repository	Stars	Language	License	Activity
1	rmax/scrapy-redisRedis-based components for Scrapy.	★ 5.6k	Python	MIT	Active
2	SpiderClub/haipproxy:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis	★ 5.5k	Python	MIT	Stale
3	celery/celeryDistributed Task Queue (development branch)	★ 29k	Python	Other	Active
4	sogou/workflowC++ Parallel Computing and Asynchronous Networking Framework	★ 14.4k	C++	Apache-2.0	Active
5	redis/redis-pyRedis Python client	★ 13.7k	Python	MIT	Active
6	rq/rqSimple job queues for Python	★ 10.7k	Python	Other	Active
7	taskforcesh/bullmqBullMQ - Message Queue and Batch processing for NodeJS, Python, .NET, Elixir, Rust and PHP based on Redis or PostgreSQL	★ 9.5k	TypeScript	MIT	Active
8	samuelclay/NewsBlurNewsBlur is a personal news reader that brings people together to talk about the world. A new sound of an old instrument.	★ 7.7k	HTML	MIT	Active