Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
β28Jul 31, 2024Updated 2 years ago
Alternatives and similar repositories for sandcrawler
Users that are interested in sandcrawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Web archive index server based on RocksDBβ46Nov 13, 2026Updated last week
- [WWW 2026] πΈ GlotWeb: Web Indexing for Minority Languagesβ17Apr 14, 2026Updated 7 months ago
- Trough: Big data, small databases.β43Jul 25, 2024Updated 2 years ago
- Parses Wikipedia citation templates in Pythonβ17Mar 26, 2025Updated last year
- EpochFS is a versioned cloud file system with git-like branching, transaction support.β17Apr 23, 2026Updated 6 months ago
- Bare Metal GPUs on DigitalOcean Gradient AI β’ AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- Converts HTTrack crawls to WARC filesβ39Aug 6, 2024Updated 2 years ago
- Perpetual Access To The Scholarly Recordβ122Jul 31, 2024Updated 2 years ago
- Load, build and explore Patstat using the Google Cloud Platformβ10Jan 19, 2019Updated 7 years ago
- Material parsers and other tools, scripts Initially developed for Grobid Superconductorβ14Feb 21, 2025Updated last year
- MatSci-LumEn: Materials Science Large Language Models Evaluation for text and data miningβ13Aug 8, 2026Updated 3 months ago
- Translation of query languages to serialized KoralQuery protocolβ15Nov 12, 2026Updated last week
- produce a stream of citiation data coming off wikimediaβ11Mar 28, 2017Updated 9 years ago
- My Docker-based setup for monitoring a Mastidin instance with Prometheusβ11Dec 8, 2022Updated 3 years ago
- Tools to analyze web archivesβ20Jul 12, 2016Updated 10 years ago
- Deploy on Railway without the complexity - Free Credits Offer β’ AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Run pkg.scripts subtasks in a runner-agnostic way (npm/yarn, whichever launched the main script)β11Dec 25, 2023Updated 2 years ago
- Small string compression using smaz compression algorithm. Fast, because it's in C. Supports Python 3+β13Nov 2, 2026Updated 2 weeks ago
- Personal Knowledge Management toolsβ10Jul 20, 2026Updated 4 months ago
- β11Oct 5, 2023Updated 3 years ago
- The EHRI project's portal interface.β15Nov 13, 2026Updated last week
- The grobidmonkey package is an open-source package designed for postprocessing GROBID outputs.β12Mar 27, 2024Updated 2 years ago
- HTTP(S) proxy that saves traffic to a WARC file, using libmitmproxy.β16Oct 25, 2013Updated 13 years ago
- Single file C header for UTF-x-to-y conversions + helpersβ13Jun 11, 2023Updated 3 years ago
- The fastest off-the-shelf inference algorithm for LLMs (ICLRβ25)β14Mar 8, 2025Updated last year
- Bare Metal GPUs on DigitalOcean Gradient AI β’ AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- Citation Classification using hybrid neural network model for Wikipedia Referencesβ31Dec 8, 2022Updated 3 years ago
- Python script to create CDX index files of WARC dataβ22Sep 4, 2025Updated last year
- A browser extension providing Open Access bibliographical servicesβ18Dec 9, 2022Updated 3 years ago
- Small n-triples to line delimited JSON converter and prefix cutter.β11Aug 5, 2019Updated 7 years ago
- A timezone converter for online eventsβ10May 10, 2020Updated 6 years ago
- ARK minter, binder, resolverβ24May 28, 2026Updated 5 months ago
- Your personal media geography. Catherine's thesis project.β10May 8, 2017Updated 9 years ago
- Specifications for better computingβ10Nov 19, 2019Updated 7 years ago
- β15Dec 18, 2023Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer β’ AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Sublime Text API Version Documenterβ11Jan 3, 2023Updated 3 years ago
- Basis of FragDenStaat.de's βKoalitionstrackerββ16Sep 4, 2026Updated 2 months ago
- Scraper for German democracy documentsβ47Sep 12, 2023Updated 3 years ago
- Berlin Chapter Pageβ11Nov 13, 2026Updated last week
- XQuery utility to create MADS/RDF (and SKOS) from MARC Authority XMLβ13Oct 17, 2024Updated 2 years ago
- StarryDigitizer is a Vue.js-based tool for extracting numeric data from plot images, supporting XY data.β14Oct 18, 2026Updated last month
- β12Dec 11, 2022Updated 3 years ago
Are these results useful?