Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki
☆28Jul 31, 2024Updated 2 years ago
Alternatives and similar repositories for sandcrawler
Users that are interested in sandcrawler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Offline wikipedia android browser☆14May 16, 2017Updated 9 years ago
- Trough: Big data, small databases.☆43Jul 25, 2024Updated 2 years ago
- search interface for scholarly works☆86Aug 2, 2024Updated 2 years ago
- Demo app built using AngularJS with Backand serving as the back end☆13Mar 1, 2017Updated 9 years ago
- Parses Wikipedia citation templates in Python☆17Mar 26, 2025Updated last year
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- EpochFS is a versioned cloud file system with git-like branching, transaction support.☆17Apr 23, 2026Updated 6 months ago
- A tool for detecting viruses and NSFW material in WARC files☆18Jul 15, 2026Updated 3 months ago
- Sort-friendly URI Reordering Transform (SURT) python module☆48Sep 11, 2025Updated last year
- Perpetual Access To The Scholarly Record☆121Jul 31, 2024Updated 2 years ago
- Converts HTTrack crawls to WARC files☆39Aug 6, 2024Updated 2 years ago
- Keeping It Simple is Hard☆11Feb 12, 2024Updated 2 years ago
- This package contains a tool for automatically cropping and deskewing images of book pages captured by an Internet Archive Scribe booksca…☆10Apr 25, 2013Updated 13 years ago
- Load, build and explore Patstat using the Google Cloud Platform☆10Jan 19, 2019Updated 7 years ago
- code and data used to build a training dataset for dragnet models☆10Nov 29, 2020Updated 5 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- Material parsers and other tools, scripts Initially developed for Grobid Superconductor☆14Feb 21, 2025Updated last year
- produce a stream of citiation data coming off wikimedia☆11Mar 28, 2017Updated 9 years ago
- A machine learning software for extracting astronomical entities from scholarly documents☆10Oct 31, 2022Updated 4 years ago
- My Docker-based setup for monitoring a Mastidin instance with Prometheus☆11Dec 8, 2022Updated 3 years ago
- Tools to analyze web archives☆20Jul 12, 2016Updated 10 years ago
- Bouton ISTEX : extension web capable d'insérer dynamiquement sur la page web consultée un lien vers le fulltext d'un document si ce dern…☆11May 30, 2023Updated 3 years ago
- Run pkg.scripts subtasks in a runner-agnostic way (npm/yarn, whichever launched the main script)☆11Dec 25, 2023Updated 2 years ago
- Official Python package for ArchiveBox, the self-hosted internet archiving solution.☆12Oct 5, 2024Updated 2 years ago
- Small string compression using smaz compression algorithm. Fast, because it's in C. Supports Python 3+☆13Updated this week
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Archiving GitHub☆11Aug 5, 2025Updated last year
- Benchmark scripts for comparing different tokenizers and sentence segmenters of German☆12Feb 27, 2023Updated 3 years ago
- Personal Knowledge Management tools☆10Jul 20, 2026Updated 3 months ago
- mirror of https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Collection.git☆10Mar 23, 2013Updated 13 years ago
- (OBSOLETE) XMLUI Language Packs (Multilingual Support / I18N) for DSpace 6.x and below☆13Jul 18, 2022Updated 4 years ago
- The grobidmonkey package is an open-source package designed for postprocessing GROBID outputs.☆12Mar 27, 2024Updated 2 years ago
- The Wikinflection Corpus, from the paper "Wikinflection Corpus: A (Better) Multilingual, Morpheme-Annotated Inflectional Corpus" (Metheni…☆12Dec 15, 2023Updated 2 years ago
- HTTP(S) proxy that saves traffic to a WARC file, using libmitmproxy.☆16Oct 25, 2013Updated 13 years ago
- Single file C header for UTF-x-to-y conversions + helpers☆13Jun 11, 2023Updated 3 years ago
- End-to-end encrypted email - Proton Mail • AdSpecial offer: 40% Off Yearly / 80% Off First Month. All Proton services are open source and independently audited for security.
- WASAPI data transfer APIs☆51Apr 23, 2022Updated 4 years ago
- Web privacy analysis of Sweden's 290 municipalities.☆11Nov 18, 2022Updated 3 years ago
- Benson turns a list of URLs into mp3s of the contents of each web page - take control over your reading backlog!☆16Oct 30, 2024Updated 2 years ago
- Specification of a stand-off element for the TEI guidelines☆12Apr 29, 2021Updated 5 years ago
- A reddit bot that finds original publish dates on linked articles.☆10Nov 30, 2024Updated last year
- A C# library for loading LSD: Dream Emulator data files.☆15Aug 28, 2023Updated 3 years ago
- Python script to create CDX index files of WARC data☆22Sep 4, 2025Updated last year