Alternatives and similar repositories for open-gov-crawlers

Users that are interested in open-gov-crawlers are comparing it to the libraries listed below

• simonw / scrape-open-data
  Scrape various open data directories to create an index of what's available out there
  ☆37Updated 4 months ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 5 years ago
• mediacloud / feed_seeker
  Find rss, atom, xml, and rdf feeds on webpages
  ☆30Updated 8 months ago
• eyeseast / feed-to-sqlite
  Save an RSS or ATOM feed to a SQLite database
  ☆52Updated 2 years ago
• public-law / datasets
  World legal info: scraped, organized, and permissively licensed under Creative Commons.
  ☆19Updated last month
• zytedata / zyte-autoextract
  Python clients for Zyte AutoExtract API
  ☆40Updated 3 years ago
• scrapinghub / scrapy-autoextract
  Zyte Automatic Extraction integration for Scrapy
  ☆56Updated 3 years ago
• reanalytics-databoutique / advanced-scrapy-proxies
  Scrapy rotation proxy package with advanced functions
  ☆95Updated 2 years ago
• niczem / trawler
  scraper for facebook, gab, google and tiktok
  ☆21Updated last week
• datamade / how-to
  📚 Doing all sorts of things, the DataMade way
  ☆98Updated 3 months ago
• appeler / clean-names
  Deduplicate and parse list of `dirty names'
  ☆23Updated 4 years ago
• xtream1101 / scraperx
  Library for scraping websites or apis at any scale
  ☆53Updated last year
• cldellow / datasette-scraper
  Add website scraping abilities to Datasette
  ☆63Updated 2 years ago
• scrapinghub / scrapinghub-stack-scrapy
  Software stack with latest Scrapy and updated deps
  ☆63Updated last week
• simonw / scrape-hacker-news-by-domain
  Scrape HN to track links from specific domains
  ☆62Updated this week
• dannguyen / local-news-data
  how hard is it to get a list of all local news sites in the United States (LOL)
  ☆8Updated 5 years ago
• jplusplus / statscraper
  A base library for building web scrapers for statistical data, and a helper ontology for (primarily Swedish) statistical data.
  ☆13Updated 4 months ago
• opensanctions / datapatch
  A Python library for defining rule-based overrides on messy data
  ☆16Updated 2 months ago
• Senzing / awesome
  Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.
  ☆59Updated 2 weeks ago
• scrapy-plugins / scrapy-zyte-api
  Zyte API integration for Scrapy
  ☆38Updated last month
• mikepqr / real-estate-scrape-eg
  A repository demonstrating the use of real-estate-scrape to store the estimated value of a property on Redfin and Zillow every night usin…
  ☆34Updated this week
• freelawproject / reporters-db
  A database of court reporters, tests and other experiments
  ☆107Updated 2 weeks ago
• johnbumgarner / newshound
  This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…
  ☆33Updated 2 years ago
• TeamHG-Memex / html-text
  Extract text from HTML
  ☆134Updated 4 years ago
• ScrapingBee / scrapy-scrapingbee
  JavaScript support and proxy rotation for Scrapy with ScrapingBee.
  ☆38Updated last year
• the-real-tokai / grablinks
  A simple and streamlined Python script to extract and filter links from a remote HTML resource.
  ☆24Updated 5 months ago
• michalgm / corpwatchapi
  The CorpWatch API uses automated parsers to extract the subsidiary relationship information from Exhibit 21 of companies' 10-K filings wi…
  ☆48Updated 4 months ago
• opensanctions / fingerprints
  Now included in rigour
  ☆151Updated last month
• scrapinghub / scrapy-poet
  Page Object pattern for Scrapy
  ☆123Updated last month
• simonw / datasette-cluster-map
  Datasette plugin that shows a map for any data with latitude/longitude columns
  ☆96Updated 10 months ago