atupal / ccrawler
A distrubuted crawler ues celery.
☆17Updated 10 years ago
Related projects ⓘ
Alternatives and complementary repositories for ccrawler
- Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations☆40Updated 6 months ago
- Search engine base (crawler, indexer and parser) using Python, Celery, RabbitMQ, CouchDB and Whoosh.☆11Updated last year
- A component that tries to avoid downloading duplicate content☆27Updated 6 years ago
- Scrapy middleware for the autologin☆37Updated 6 years ago
- Find which links on a web page are pagination links☆29Updated 7 years ago
- [not actively maintained] The C++ webkit-server from capybara-webkit with useful extensions and Python bindings☆48Updated 3 years ago
- ☆32Updated 10 months ago
- Restrict crawl and scraping scope using matchers.☆25Updated 8 years ago
- Cubes OLAP Examples☆76Updated 6 years ago
- ☆18Updated 8 years ago
- Scrapy middleware which allows to crawl only new content☆79Updated 2 years ago
- Traptor -- A distributed Twitter feed☆26Updated 2 years ago
- [UNMAINTAINED] Deploy, run and monitor your Scrapy spiders.☆11Updated 9 years ago
- A middleware to use random user agent in Scrapy crawler.☆33Updated 11 years ago
- Scan for missing timeout calls in python source files☆18Updated 4 years ago
- Turn your IPython console into a cross-database SQL client☆31Updated 8 years ago
- [UNMAINTAINED] Firefox addon for Scrapely☆5Updated 8 years ago
- Jabba's headless webkit browser for scraping AJAX-powered webpages.☆91Updated 10 years ago
- Simple arithmetic expression parser☆37Updated 2 years ago
- Python Domain Specific Language Tools☆83Updated 2 years ago
- Site Hound (previously THH) is a Domain Discovery Tool☆23Updated 3 years ago
- extract difference between two html pages☆32Updated 6 years ago
- Gevent Crawling in Python, with Utilities☆23Updated 9 years ago
- Dynamic data analysis over the web. The logic to your data dashboards.☆156Updated 9 years ago
- Drop-in replacement of Django admin comes with lots of goodies, fully extensible with plugin support, pretty UI based on Twitter Bootstra…☆9Updated 6 months ago
- A Python wrapper for MADlib(http://madlib.net) - an open source library for scalable in-database machine learning algorithms☆63Updated 4 years ago
- ctypes bindings for libphash to robustly compare media files☆12Updated last year
- Stream processing in Python of twitter searches using public APIs.☆9Updated 8 years ago
- Tool to flatten stream of JSON-like objects, configured via schema☆33Updated 5 years ago