Alternatives and similar repositories for web-languages:

Users that are interested in web-languages are comparing it to the libraries listed below

• r-three / common-pile
  Repo to hold code and track issues for the collection of permissively licensed data
  ☆23Updated this week
• bltlab / paranames
  ParaNames: A multilingual resource for parallel names
  ☆31Updated 10 months ago
• bigscience-workshop / lam
  Libraries, Archives and Museums (LAM)
  ☆82Updated 2 years ago
• malteos / llm-datasets
  A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
  ☆56Updated 8 months ago
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆21Updated last year
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆24Updated 4 months ago
• EuroEval / EuroEval
  The robust European language model benchmark.
  ☆94Updated this week
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆36Updated last week
• laurieburchell / open-lid-dataset
  Repository accompanying "An Open Dataset and Model for Language Identification" (Burchell et al., 2023)
  ☆70Updated 11 months ago
• mainlp / germanic-lrl-corpora
  A survey of corpora for Germanic low-resource languages and dialects
  ☆25Updated 3 months ago
• mbanon / fastspell
  Targetted language identifier, based on FastText and Hunspell.
  ☆34Updated last month
• oscar-project / ungoliant
  The pipeline for the OSCAR corpus
  ☆167Updated last year
• cisnlp / GlotLID
  💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023
  ☆124Updated 4 months ago
• cisnlp / Glot500
  Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023
  ☆100Updated 11 months ago
• UBC-NLP / afrolid
  AfroLID, a powerful neural toolkit for African languages identification which covers 517 African languages.
  ☆31Updated 3 weeks ago
• opinionscience / BERTransfer
  A BERT-based application for reusable text classification at scale
  ☆38Updated last year
• ltgoslo / ltg-bert
  LTG-Bert
  ☆31Updated last year
• cisnlp / GlotCC
  🕸 GlotCC Dataset and Pipline -- NeurIPS 2024
  ☆18Updated 5 months ago
• timpal0l / ScandiSent
  Sentiment Corpus for Swedish 🇸🇪 Norwegian 🇳🇴 Danish 🇩🇰 Finnish 🇫🇮 (and English 🏴󠁧󠁢󠁥󠁮󠁧󠁿)
  ☆15Updated 3 years ago
• johko / awesome-german-open-source-ml
  A list of awesome open source projects in the machine learning field, who's developers are mainly based in Germany
  ☆42Updated 6 months ago
• flairNLP / CleanCoNLL
  The CleanCoNLL dataset from our EMNLP 2023 paper where we corrected annotation errors and inconsistencies in CoNLL-03.
  ☆23Updated 9 months ago
• Babelscape / wikineural
  Data and evaluation code for the paper WikiNEuRal: Combined Neural and Knowledge-based Silver Data Creation for Multilingual NER (EMNLP 2…
  ☆67Updated 2 years ago
• chandar-lab / NeoBERT
  ☆43Updated last month
• davidberenstein1957 / spacy-setfit
  This repository contains an easy and intuitive approach to use SetFit in combination with spaCy.
  ☆78Updated last year
• bigscience-workshop / data_tooling
  Tools for managing datasets for governance and training.
  ☆83Updated 2 months ago
• konstantinjdobler / focus
  [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"
  ☆30Updated 5 months ago
• stefan-it / modern-bert-ner
  My NER Experiments with ModernBERT
  ☆18Updated 2 months ago
• neuml / staticvectors
  🔢 Work with static vector models
  ☆23Updated 2 months ago
• salesforce / simplification
  ☆21Updated 2 months ago