alexksikes / mass-scraping

Related projects ⓘ

Alternatives and complementary repositories for mass-scraping

• openeventdata / scraper
  Scrapes sites. Gets news. Eventually events.
  ☆82Updated 8 years ago
• parul1931 / walmart
  scraping from walmart, target and homedepot website and getting data from amazon api
  ☆15Updated 8 years ago
• sensiblecodeio / scraperwiki-python
  ScraperWiki Python library for scraping and saving data
  ☆159Updated last year
• tpeng / googlesearch
  Scrape the Google search result with Scrapy.
  ☆98Updated 4 years ago
• gjreda / craigslist-checker
  Send text when a new Craigslist posting matches a given keyword or phrase
  ☆96Updated 9 years ago
• dcondrey / scrapy-spiders
  Collection of python scripts I have created to crawl various websites, mostly for lead generation projects to match keywords and collect …
  ☆129Updated last year
• pudo-attic / scrapekit
  Python library with common functionality for writing web scrapers
  ☆102Updated 9 years ago
• benjaminestes / pyscape-client
  A library to interface with the Linkscape API.
  ☆41Updated 6 years ago
• madmatah / seo-ranking-engine
  SEO Tool to track ranking of keywords on search engines (google app engine application)
  ☆47Updated 12 years ago
• HyperionGray / starbelly
  Streaming web crawler with WebSocket API
  ☆44Updated last year
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆57Updated 9 months ago
• cldellow / real-estate-prices-cc
  Source real estate prices from the Common Crawl.
  ☆27Updated 6 years ago
• TeamHG-Memex / fortia
  [UNMAINTAINED] Firefox addon for Scrapely
  ☆5Updated 8 years ago
• dannguyen / facebook-trending-rss-fetcher
  Python code to scrape and collect data from the RSS feeds Facebook uses to augment its Trending Section
  ☆57Updated 6 years ago
• NUKnightLab / twxplorer
  ☆36Updated last year
• nikoma / social_media_monitoring
  Social media monitoring tools such as sentiment analysis, keyword tracking and more
  ☆46Updated 10 years ago
• CNuge / email-report
  A modular template for scraping data from the web to send yourself scheduled email reports
  ☆40Updated 4 years ago
• openstates / billy
  legacy backend for Open States
  ☆87Updated 4 years ago
• duydo / scrapy-crunchbase
  Using Scrapy to get company profiles from http://crunchbase.com
  ☆31Updated 11 years ago
• storerjeremy / python-semrush
  Python wrapper around the SEMrush API.
  ☆30Updated last year
• nicodjimenez / bus_catchers
  Python scripts for scraping bus ticket data from the websites of BoltBus, Greyhound, Megabus, GoBus, Amtrak, Peterpan, and EasternTravel.
  ☆39Updated 4 years ago
• robertoarruda / linkedin-public-dir-companies
  Crawler and scraper of the public directory of companies on LinkedIn.
  ☆25Updated 5 years ago
• odie5533 / WarcMiddleware
  WarcMiddleware lets users seamlessly download a mirror copy of a website when running a web crawl with the Python web crawler Scrapy.
  ☆44Updated 6 years ago
• TeamHG-Memex / sitehound-frontend
  Site Hound (previously THH) is a Domain Discovery Tool
  ☆23Updated 3 years ago
• PhantomInsights / forhirehelper
  A Python desktop application that makes the use of freelancing subreddits easier and faster.
  ☆35Updated 4 years ago
• scrapinghub / exporters
  Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
  ☆40Updated 6 months ago
• ianozsvald / twitter_networkx_concept_map
  Take streaming tweets, extract hashtags & usernames, create graph, export graphml for Gephi visualisation
  ☆33Updated 11 years ago
• jorgenkg / python-proxy-rotator
  An automatic proxy rotator - multithreaded & SSL
  ☆78Updated 3 years ago