GateNLP / ultimate-sitemap-parser
Ultimate Website Sitemap Parser
☆190Updated this week
Alternatives and similar repositories for ultimate-sitemap-parser:
Users that are interested in ultimate-sitemap-parser are comparing it to the libraries listed below
- Modern robots.txt Parser for Python☆189Updated last year
- Extract text from HTML☆133Updated 4 years ago
- Fast and robust date extraction from web pages, with Python or on the command-line☆121Updated 3 weeks ago
- Web scraping Page Objects core library☆96Updated 3 months ago
- Python port of Boilerpipe library☆86Updated 5 months ago
- Extract price amount and currency symbol from a raw text string☆318Updated 2 months ago
- Detect and classify pagination links☆100Updated 4 years ago
- Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.☆148Updated this week
- A pure-Python robots.txt parser with support for modern conventions.☆57Updated this week
- Parsing JavaScript objects into Python data structures☆201Updated 3 weeks ago
- A Scrapy middleware to bypass the CloudFlare's anti-bot protection☆105Updated 3 years ago
- Automatic unit test generation for Scrapy.☆55Updated 3 years ago
- Software stack with latest Scrapy and updated deps☆63Updated 3 weeks ago
- The most advanced debugging and testing tool for Scrapy☆16Updated last year
- Page Object pattern for Scrapy☆119Updated this week
- Common interface for data container classes☆62Updated this week
- Article extraction benchmark: dataset and evaluation scripts☆300Updated 9 months ago
- Simple, robust email validation☆129Updated 2 years ago
- A python based HTML to text conversion library, command line client and Web service.☆282Updated last week
- A helper library full of URL-related heuristics.☆64Updated 3 months ago
- Library to populate items using XPath and CSS with a convenient API☆45Updated 3 months ago
- Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters☆133Updated 3 weeks ago
- URL normalization for Python☆94Updated 2 years ago
- Zyte API integration for Scrapy☆37Updated last week
- Python clients for Zyte AutoExtract API☆40Updated 3 years ago
- Parse numbers written in natural language☆109Updated 3 months ago
- admin ui for scrapy/open source scrapinghub☆58Updated 3 years ago
- This repository provides usage examples for the Python module Newspaper3k.☆144Updated last year
- CoCrawler is a versatile web crawler built using modern tools and concurrency.☆189Updated 2 years ago
- Pick the most common user-agents on the Internet 👻☆170Updated 4 years ago