ilinguistics / common_crawl_corpusLinks
Scripts for building a geo-located web corpus using Common Crawl data
☆11Updated 2 months ago
Alternatives and similar repositories for common_crawl_corpus
Users that are interested in common_crawl_corpus are comparing it to the libraries listed below
Sorting:
- Crawling engine that crawls a set of top-level domains looking for documents in a list of languages☆11Updated last year
- 🕸 GlotWeb: Web Indexing for Low-Resource Languages -- under construction.☆17Updated 4 months ago
- Harassment Lexicon and Corpus☆30Updated 7 years ago
- DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…☆52Updated 5 years ago
- Bilingual sengence aligner☆28Updated last month
- Toolkit to compile a comparable/parallel corpus from European Parliament proceedings☆16Updated 5 years ago
- Language Tool style grammar handling with spaCy 2.0☆42Updated 7 years ago
- A library to extract a publication date from a web page, along with a measure of the accuracy.☆41Updated 6 years ago
- Unsupervised method for extracting quotation-speaker pairs from large news corpora.☆29Updated 7 years ago
- A collection of over 1.5 Million tweets data translated to French, with their sentiment.☆35Updated 8 years ago
- ADEL is a robust and efficient entity linking framework that is adaptive to text genres and language, entity types for the classification…☆19Updated 6 years ago
- Automated paraphrases Generation☆36Updated 3 years ago
- Corpus of Attribution-Annotated news articles covering the campaigns during the year leading up to the 2016 US Presidential election.☆20Updated 7 years ago
- Data collection, alignment and TAUS repository☆23Updated 8 years ago
- Use BERT to Fill in the Blanks☆84Updated 4 years ago
- Measure the readability of a given text using surface characteristics☆81Updated 11 months ago
- BERT models for many languages created from Wikipedia texts☆33Updated 5 years ago
- Röttger et al. (WOAH at NAACL 2022): "Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models"☆17Updated 3 years ago
- A Named-Entity Recogniser based on Grobid.☆54Updated 7 months ago
- MinScIE is an Open Information Extraction system which provides structured knowledge enriched with semantic information about citations.☆15Updated 6 years ago
- SciWING is a modern toolkit for scientific document processing from WING-NUS☆63Updated 2 years ago
- ParaNames: A multilingual resource for parallel names☆39Updated last year
- An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.☆86Updated 4 years ago
- ☆54Updated 4 years ago
- An implementation of GrASP (Shnarch et. al., 2017)☆23Updated 3 years ago
- An example of how to use spaCy for extremely large files without running into memory issues☆36Updated 3 years ago
- ☆18Updated 4 years ago
- Code accompanying the submission "Structural Text Segmentation of Legal Documents" by Aumiller et al.☆98Updated 2 years ago
- Code and data used in named entity transliteration experiments☆57Updated 7 years ago
- Arabic News Stance Corpus☆11Updated 4 years ago