Alternatives and similar repositories for GlotLID

Users that are interested in GlotLID are comparing it to the libraries listed below

• cisnlp / GlotWeb

  View on GitHub
  🕸 GlotWeb: Web Indexing for Low-Resource Languages -- under construction.
  ☆17Aug 13, 2025Updated 6 months ago
• laurieburchell / open-lid-dataset

  View on GitHub
  Repository accompanying "An Open Dataset and Model for Language Identification" (Burchell et al., 2023)
  ☆74Apr 1, 2025Updated 10 months ago
• cisnlp / ofa

  View on GitHub
  A Framework aims to wisely initialize unseen subword embeddings in PLMs for efficient large-scale continued pretraining
  ☆18Nov 26, 2023Updated 2 years ago
• kensho-technologies / pathpiece

  View on GitHub
  PathPiece tokenizer
  ☆13Nov 10, 2024Updated last year
• cisnlp / Glot500

  View on GitHub
  Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023
  ☆106Apr 20, 2024Updated last year
• fyvo / WMT-Biomed-Test

  View on GitHub
  ☆13Aug 23, 2024Updated last year
• Helsinki-NLP / OpusFilter

  View on GitHub
  OpusFilter - Parallel corpus processing toolkit
  ☆115Updated this week
• cisnlp / mPLM-Sim

  View on GitHub
  mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
  ☆11Jan 19, 2024Updated 2 years ago
• lucy3 / whos_filtered

  View on GitHub
  ☆14Oct 4, 2024Updated last year
• huggingface / datatrove

  View on GitHub
  Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
  ☆2,885Updated this week
• bitextor / bifixer

  View on GitHub
  Tool to fix bitexts and tag near-duplicates for removal
  ☆34Sep 4, 2025Updated 5 months ago
• huggingface / fineweb-2

  View on GitHub
  ☆220Oct 27, 2025Updated 3 months ago
• IST-DASLab / qmoe

  View on GitHub
  Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".
  ☆279Nov 3, 2023Updated 2 years ago
• masakhane-io / masakhane-pos

  View on GitHub
  POS for African languages
  ☆19Jun 25, 2025Updated 7 months ago
• konstantinjdobler / focus

  View on GitHub
  [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"
  ☆36Jun 7, 2025Updated 8 months ago
• ZurichNLP / mbr

  View on GitHub
  Minimum Bayes Risk Decoding for Hugging Face Transformers
  ☆60Jun 3, 2024Updated last year
• MicrosoftTranslator / NTREX

  View on GitHub
  NTREX -- News Test References for MT Evaluation
  ☆88Jun 5, 2024Updated last year
• Aleph-Alpha-Research / trigrams

  View on GitHub
  ☆59Nov 18, 2025Updated 2 months ago
• mainlp / germanic-lrl-corpora

  View on GitHub
  Overview of corpora/datasets for Germanic low-resource languages and dialects. Accompanies "A Survey of Corpora for Germanic Low-Resource…
  ☆27Updated this week
• CPJKU / wechsel

  View on GitHub
  Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.
  ☆87Sep 12, 2024Updated last year
• juletx / self-translate

  View on GitHub
  Do Multilingual Language Models Think Better in English?
  ☆42Aug 3, 2023Updated 2 years ago
• malteos / llm-datasets

  View on GitHub
  A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
  ☆64Jul 29, 2024Updated last year
• langtech-bsc / mt-evaluation

  View on GitHub
  A framework for evaluating Machine Translation models.
  ☆12May 26, 2025Updated 8 months ago
• ait-cs-IaaS / Taranis-NG

  View on GitHub
  Taranis NG is an OSINT gathering and analysis tool for CSIRT teams and organisations. It allows team-to-team collaboration, and contains …
  ☆10Oct 17, 2023Updated 2 years ago
• dadelani / sib-200

  View on GitHub
  SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
  ☆23Jan 26, 2025Updated last year
• thammegowda / mtdata

  View on GitHub
  A tool that locates, downloads, and extracts machine translation corpora
  ☆162Sep 18, 2025Updated 4 months ago
• huggingface / olm-datasets

  View on GitHub
  Pipeline for pulling and processing online language model pretraining data from the web
  ☆176Jul 31, 2023Updated 2 years ago
• Helsinki-NLP / mammoth

  View on GitHub
  MAMMOTH: MAssively Multilingual Modular Open Translation @ Helsinki
  ☆30Updated this week
• facebookresearch / flores

  View on GitHub
  Facebook Low Resource (FLoRes) MT Benchmark
  ☆762Nov 20, 2023Updated 2 years ago
• facebookresearch / belebele

  View on GitHub
  Repo for the Belebele dataset, a massively multilingual reading comprehension dataset.
  ☆340Dec 18, 2024Updated last year
• conradj / pocket-public-archive

  View on GitHub
  statically generated weekly digest of articles read in Pocket
  ☆10May 14, 2019Updated 6 years ago
• commoncrawl / ia-web-commons

  View on GitHub
  Web archiving utility library
  ☆11Dec 3, 2025Updated 2 months ago
• eryk-mazus / sigh

  View on GitHub
  Seamless Voice Interactions with LLMs
  ☆12Oct 28, 2023Updated 2 years ago
• scandale-project / scandale

  View on GitHub
  A libre software which is providing a backend architecture for collecting data from probes and storing proof of checks.
  ☆11Jan 16, 2026Updated 3 weeks ago
• jiangycTarheel / SQ-Transformer

  View on GitHub
  ☆10Feb 12, 2024Updated 2 years ago
• Chocapikk / CVE-2023-46805

  View on GitHub
  Ivanti Pulse Secure CVE-2023-46805 Scanner - Based on Assetnote's Research
  ☆12Jan 19, 2024Updated 2 years ago
• zhuyjan / MER2025-MRAC25

  View on GitHub
  [ACM-MM 2025 Workshop] More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignment.
  ☆25Nov 25, 2025Updated 2 months ago
• google-research / url-nlp

  View on GitHub
  ☆263Aug 1, 2025Updated 6 months ago
• ariG23498 / RNN_Viz

  View on GitHub
  Sequence models in Numpy
  ☆25Oct 9, 2020Updated 5 years ago