tballison / commoncrawl-fetcher-liteLinks
Simplified version of a common crawl fetcher
☆17Updated last week
Alternatives and similar repositories for commoncrawl-fetcher-lite
Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below
Sorting:
- Template repository and README for submissions to Bellingcat's Global Hackathon☆16Updated 3 years ago
- RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter☆20Updated 3 years ago
- A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…☆72Updated 2 weeks ago
- Common Crawl Index Server☆71Updated 9 months ago
- An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)☆25Updated 8 years ago
- An open source investigation tool to collect and analyse public VK community wall posts☆35Updated 3 years ago
- An index of PDF-centric corpora☆149Updated 5 months ago
- A search interface and wayback machine for the UKWA Solr based warc-indexer framework.☆132Updated last week
- Napkin is a simple tool to produce statistical analysis of a text☆12Updated last year
- Advanced desktop search/corpus exploration prototype☆21Updated 4 years ago
- Capture a URL with Playwright☆30Updated this week
- An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…☆154Updated 2 months ago
- Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs☆21Updated 3 months ago
- A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine☆189Updated 3 weeks ago
- ☆20Updated last year
- DEPRECATED. Desktop graph visualization application☆51Updated 3 years ago
- Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.☆66Updated last week
- metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)☆34Updated last month
- CSIRT Tooling: Best Practices in Developing, Maintaining and Distributing Open Source Tools☆16Updated 3 years ago
- API client for Aleph, supports bulk entity and document upload.☆28Updated last year
- External twitter feeder for AIL framework☆16Updated 2 years ago
- Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…☆131Updated 3 weeks ago
- Similarius is a Python library to compare web page and evaluate the level of similarity.☆22Updated last week
- Extract Unique Word Lists From Wikipedia Database☆13Updated 5 years ago
- A cross-platform command line tool for parallelised content extraction and analysis.☆248Updated last month
- Analyze and help extract older "hidden" versions of a pdf from the current pdf.☆87Updated 3 years ago
- Granular Viewer of Sentiments Between Entities in Massively Large Documents and Collections of Texts, powered by AREkit☆38Updated last week
- A tool to detect whether a PDF has a bad redaction☆159Updated this week
- You're busted!☆27Updated 5 years ago
- This database tracks details about victims of NSO Group’s Pegasus spyware, based on reporting by Amnesty International, Citizen Lab, Forb…☆34Updated 2 years ago