Alternatives and similar repositories for common_crawl_corpus

Users that are interested in common_crawl_corpus are comparing it to the libraries listed below

• transducens / linguacrawl
  Crawling engine that crawls a set of top-level domains looking for documents in a list of languages
  ☆11Updated last year
• cisnlp / GlotWeb
  🕸 GlotWeb: Web Indexing for Low-Resource Languages -- under construction.
  ☆17Updated 5 months ago
• BramVanroy / spacy-extreme
  An example of how to use spaCy for extremely large files without running into memory issues
  ☆36Updated 3 years ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆52Updated 5 years ago
• tokestermw / spacy_grammar
  Language Tool style grammar handling with spaCy 2.0
  ☆42Updated 7 years ago
• rewire-online / multilingual-hatecheck
  Röttger et al. (WOAH at NAACL 2022): "Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models"
  ☆17Updated 3 years ago
• writer / fitbert
  Use BERT to Fill in the Blanks
  ☆84Updated 4 years ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 6 years ago
• gamebusterz / French-Sentiment-Analysis-Dataset
  A collection of over 1.5 Million tweets data translated to French, with their sentiment.
  ☆35Updated 8 years ago
• chozelinek / europarl
  Toolkit to compile a comparable/parallel corpus from European Parliament proceedings
  ☆16Updated 6 years ago
• AudayBerro / automatedParaphrase
  Automated paraphrases Generation
  ☆36Updated 3 years ago
• zyocum / dedup
  Find duplicate text files.
  ☆15Updated last year
• MaLA-LM / GlotEval
  GlotEval: a unified evaluation toolkit designed to benchmark multilingual Large Language Models (LLMs) in a language-specific way
  ☆18Updated 2 months ago
• apepa / claim-rank
  ☆16Updated 7 years ago
• loomchild / maligna
  Bilingual sengence aligner
  ☆28Updated 2 months ago
• Mrezvan94 / Harassment-Corpus
  Harassment Lexicon and Corpus
  ☆30Updated 7 years ago
• TurkuNLP / wikibert
  BERT models for many languages created from Wikipedia texts
  ☆33Updated 5 years ago
• plkumjorn / GrASP
  An implementation of GrASP (Shnarch et. al., 2017)
  ☆23Updated 3 years ago
• andreasvc / readability
  Measure the readability of a given text using surface characteristics
  ☆81Updated last year
• akb89 / pyfn
  A python module to process data for Frame Semantic Parsing
  ☆23Updated 5 years ago
• CogComp / talen
  A way to do annotations for NER. TALEN: Tool for Annotation of Low-resource ENtities
  ☆118Updated 6 months ago
• ziyin-dl / ngram-word2vec
  ☆18Updated 4 years ago
• mayhewsw / pytorch-truecaser
  A simple neural truecaser written in pytorch and allennlp.
  ☆33Updated last year
• nelson-liu / lexical-semantic-recognition
  ☆18Updated 2 years ago
• ruanchaves / hashformers
  Accurate word segmentation for hashtags and text, powered by Transformers and Beam Search. A scalable alternative to heuristic splitters …
  ☆76Updated 3 weeks ago
• modernmt / DataCollection
  Data collection, alignment and TAUS repository
  ☆23Updated 8 years ago
• LoLei / redditcleaner
  Cleans Reddit Text Data
  ☆84Updated 5 years ago
• matatusko / opinion-or-fact-sentence-classifier
  Classifies sentences whether they represent a fact or personal opinion with 90% accuracy using various Machine Learning algorithms from s…
  ☆31Updated 7 years ago
• ljos / navnkjenner
  Named-Entity Recognition for Norwegian Bokmål and Nynorsk
  ☆12Updated 6 years ago
• grobidOrg / grobid-ner
  A Named-Entity Recogniser based on Grobid.
  ☆54Updated 8 months ago