Alternatives and similar repositories for open-gov-crawlers

Users that are interested in open-gov-crawlers are comparing it to the libraries listed below

• medialab / ural
  A helper library full of URL-related heuristics.
  ☆72Updated 2 months ago
• scrapinghub / web-poet
  Web scraping Page Objects core library
  ☆103Updated 2 weeks ago
• scrapinghub / scrapinghub-stack-scrapy
  Software stack with latest Scrapy and updated deps
  ☆65Updated 4 months ago
• opensanctions / fingerprints
  Now included in rigour
  ☆152Updated 2 weeks ago
• stav / wgrep
  Web grep: search all rendered resources used by a URI
  ☆89Updated 2 weeks ago
• dogsheep / google-takeout-to-sqlite
  Save data from Google Takeout to a SQLite database
  ☆117Updated 2 years ago
• TeamHG-Memex / html-text
  Extract text from HTML
  ☆135Updated 5 years ago
• scrapinghub / scrapy-poet
  Page Object pattern for Scrapy
  ☆124Updated last month
• zytedata / zyte-autoextract
  Python clients for Zyte AutoExtract API
  ☆41Updated 3 years ago
• scrapinghub / scrapy-autoextract
  Zyte Automatic Extraction integration for Scrapy
  ☆56Updated 3 years ago
• opensanctions / yente
  API for OpenSanctions with support for entity search and bulk matching of data collections. Supports Reconciliation API spec.
  ☆115Updated this week
• cldellow / datasette-scraper
  Add website scraping abilities to Datasette
  ☆66Updated 2 years ago
• zytedata / zyte-spider-templates
  Spider templates for automatic crawlers.
  ☆32Updated 2 months ago
• TeamHG-Memex / scrapy-crawl-once
  Scrapy middleware which allows to crawl only new content
  ☆79Updated 3 years ago
• freelawproject / reporters-db
  A database of court reporters, tests and other experiments
  ☆117Updated 2 weeks ago
• simonw / scrape-open-data
  Scrape various open data directories to create an index of what's available out there
  ☆37Updated 9 months ago
• public-law / datasets
  World legal info: scraped, organized, and permissively licensed under Creative Commons.
  ☆20Updated 6 months ago
• andrewtavis / wikirepo
  Python based Wikidata framework for easy dataframe extraction
  ☆45Updated 2 years ago
• reanalytics-databoutique / advanced-scrapy-proxies
  Scrapy rotation proxy package with advanced functions
  ☆95Updated 3 years ago
• Senzing / awesome
  Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.
  ☆63Updated this week
• palewire / savepagenow
  A simple Python wrapper and command-line interface for archive.org’s "Save Page Now" capturing service
  ☆188Updated last year
• eyeseast / feed-to-sqlite
  Save an RSS or ATOM feed to a SQLite database
  ☆57Updated last month
• dogsheep / pocket-to-sqlite
  Create a SQLite database containing data from your Pocket account
  ☆107Updated 2 years ago
• peopledatalabs / peopledatalabs-python
  A Python client for the People Data Labs API
  ☆36Updated this week
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆191Updated 3 years ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆142Updated last month
• jamesturk / spatula
  A modern Python library for writing maintainable web scrapers.
  ☆248Updated 2 weeks ago
• scrapy / itemadapter
  Common interface for data container classes
  ☆68Updated last month
• openschemas / schemaorg
  python functions for applied use of schema.org
  ☆36Updated 4 years ago
• mscarey / AuthoritySpoke
  Reading legal authority for the last time
  ☆41Updated 9 months ago