scrapinghub / adblockparser

Related projects: ⓘ

• joepie91 / python-whois
  A python module for retrieving and parsing WHOIS data
  ☆398Updated 3 years ago
• seomoz / reppy
  Modern robots.txt Parser for Python
  ☆185Updated 8 months ago
• TeamHG-Memex / autologin
  A project to attempt to automatically login to a website given a single seed
  ☆122Updated 2 years ago
• TeamHG-Memex / Formasaurus
  Formasaurus tells you the type of an HTML form and its fields using machine learning
  ☆116Updated 3 months ago
• ko-zu / psl
  publicsuffixlist for python
  ☆61Updated last week
• TeamHG-Memex / autopager
  Detect and classify pagination links
  ☆98Updated 4 years ago
• scrapinghub / shub
  Scrapinghub Command Line Client
  ☆125Updated 4 months ago
• redapple / parslepy
  Python implementation of the Parsley language for extracting structured data from web pages
  ☆92Updated 6 years ago
• rbaier / python-urltools
  ☆78Updated this week
• scrapy-plugins / scrapy-deltafetch
  Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
  ☆264Updated 2 years ago
• scrapy-plugins / scrapy-magicfields
  Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
  ☆56Updated 2 years ago
• seomoz / pyreBloom
  Fast Redis Bloom Filters in Python
  ☆289Updated 5 years ago
• TeamHG-Memex / aquarium
  Splash + HAProxy + Docker Compose
  ☆196Updated 5 years ago
• seomoz / url-py
  URL Transformation, Sanitization
  ☆103Updated 8 months ago
• barseghyanartur / tld
  Extracts the top level domain (TLD) from the URL given.
  ☆177Updated last year
• chazkii / chromewhip
  Scriptable Google Chrome™ as a HTTP service + asyncio driver
  ☆118Updated last year
• TeamHG-Memex / undercrawler
  A generic crawler
  ☆78Updated 6 years ago
• habnabit / txsocksx
  SOCKS{4,4a,5} endpoints for twisted
  ☆59Updated 4 years ago
• scrapinghub / python-scrapinghub
  A client interface for Scrapinghub's API
  ☆202Updated 8 months ago
• brandicted / scrapy-webdriver
  ☆144Updated 8 years ago
• lorien / user_agent
  Generator of User-Agent header
  ☆335Updated 2 months ago
• scrapy / w3lib
  Python library of web-related functions
  ☆388Updated 3 months ago
• TeamHG-Memex / arachnado
  Web Crawling UI and HTTP API, based on Scrapy and Tornado
  ☆161Updated last year
• axiak / pyre2
  Python wrapper for RE2
  ☆295Updated last year
• roycehaynes / scrapy-rabbitmq
  A RabbitMQ Scheduler for Scrapy
  ☆85Updated 2 years ago
• bookieio / breadability
  Reworked https://www.readability.com/ parsing library (now https://mercury.postlight.com/ is living alternative)
  ☆204Updated 4 months ago
• scrapinghub / webpager
  Paginating the web
  ☆37Updated 10 years ago
• GateNLP / ultimate-sitemap-parser
  Ultimate Website Sitemap Parser
  ☆178Updated last year
• scrapy / cssselect
  CSS Selectors for Python
  ☆291Updated 5 months ago
• michaelhelmick / lassie
  Web Content Retrieval for Humans™
  ☆611Updated 2 years ago