jonathandunn / common_crawl_corpus
Scripts for building a geo-located web corpus using Common Crawl data
☆11Updated this week
Alternatives and similar repositories for common_crawl_corpus:
Users that are interested in common_crawl_corpus are comparing it to the libraries listed below
- Crawling engine that crawls a set of top-level domains looking for documents in a list of languages☆10Updated last year
- An example of how to use spaCy for extremely large files without running into memory issues☆36Updated 2 years ago
- Finds linguistic patterns effortlessly☆36Updated last year
- Analyze Argumentation and Rhetorical Aspects in Scientific Writing.☆19Updated 2 years ago
- DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…☆52Updated 4 years ago
- Searching in-memory corpus with Corpus Query Language (CQL)☆19Updated 4 months ago
- Data collection, alignment and TAUS repository☆23Updated 7 years ago
- several algorithms for converting dependency structures into constituency structures.☆10Updated 3 years ago
- Multilingual Language Modeling Toolkit☆11Updated 7 years ago
- ☆17Updated 3 years ago
- An implementation of GrASP (Shnarch et. al., 2017)☆21Updated 2 years ago
- BERT models for many languages created from Wikipedia texts☆33Updated 4 years ago
- This repository includes all the code and data for the paper ELiDi (End2end Entity Linking and Disambiguation)☆14Updated 3 years ago
- Multilingual Open Text☆25Updated 5 months ago
- Converter from UD-trees to BART representation☆36Updated last year
- GC4LM: A Colossal (Biased) language model for German☆13Updated 3 years ago
- A python library to generate highly realistic typos (fuzz-testing)☆11Updated last month
- Lexicons for the Multilingual UCREL Semantic Analysis System☆41Updated last year
- Python Multilingual Ucrel Semantic Analysis System☆31Updated 8 months ago
- These are lists for a variety of languages containing words that are distinctive to each language.☆38Updated 3 years ago
- LeBLEU: Levenshtein/Letter-edit BLEU, N-gram-based Translation Evaluation Score for Morphologically Complex Languages☆10Updated 4 years ago
- A Named-Entity Recogniser based on Grobid.☆52Updated 7 months ago
- Polyglot is a language identifier for detecting text documents containing text written in more than one language, and for identifying the…☆33Updated 8 years ago
- LNEx: Location Name Extractor☆25Updated 4 years ago
- A Super-Lightweight Annotation Tool for Experts: Label text in a terminal with just Python☆101Updated 4 months ago
- List of corpora annotated for coreference for different languages☆17Updated 8 months ago
- linguistic converter / merging tool for multi-level annotated corpora. graph-based (using Python and NetworkX).☆51Updated 2 years ago
- OpenNeuroSpell contains parts of NeuroSpell (http://neurospell.com/en.php) released as open-source. More code will be published as soon a…☆20Updated 5 months ago
- an experimental implementation of Burrow's delta in Python 3☆21Updated 3 years ago
- Many Natural Language Processing tasks rely on sentence boundary detection (SBD). Although amazing libraries like spacy provide state of …☆61Updated 4 years ago