Alternatives and similar repositories for web-languages

Users that are interested in web-languages are comparing it to the libraries listed below

• cisnlp / GlotLID
  💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023
  ☆138Updated 2 weeks ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆79Updated 3 weeks ago
• google-research / metricx
  ☆96Updated 6 months ago
• neuml / staticvectors
  🔢 Work with static vector models
  ☆28Updated 2 months ago
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆23Updated last year
• Pleias / marginalia
  ☆67Updated last year
• MoritzLaurer / prompt_templates
  A library for working with prompt templates locally or on the Hugging Face Hub.
  ☆46Updated 3 months ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆25Updated 6 months ago
• bitextor / warc2text
  Extracts plain text, language identification and more metadata from WARC records
  ☆22Updated 3 months ago
• malteos / llm-datasets
  A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
  ☆59Updated 10 months ago
• Knowledgator / FlashDeBERTa
  Trully flash implementation of DeBERTa disentangled attention mechanism.
  ☆58Updated last month
• davanstrien / haiku-dpo
  Using open source LLMs to build synthetic datasets for direct preference optimization
  ☆64Updated last year
• chandar-lab / NeoBERT
  ☆56Updated 3 weeks ago
• Knowledgator / GLiClass
  Generalist and Lightweight Model for Text Classification
  ☆133Updated last week
• laurieburchell / open-lid-dataset
  Repository accompanying "An Open Dataset and Model for Language Identification" (Burchell et al., 2023)
  ☆74Updated 2 months ago
• machinetranslate / machinetranslate.org
  Open information and community for machine translation
  ☆78Updated last week
• bltlab / paranames
  ParaNames: A multilingual resource for parallel names
  ☆34Updated last year
• flairNLP / transformer-ranker
  Efficiently find the best-suited language model (LM) for your NLP task
  ☆124Updated 2 weeks ago
• bigscience-workshop / lam
  Libraries, Archives and Museums (LAM)
  ☆84Updated 2 years ago
• EuroEval / EuroEval
  The robust European language model benchmark.
  ☆104Updated last week
• flairNLP / fabricator
  [EMNLP 2023 Demo] fabricator - annotating and generating datasets with large language models.
  ☆109Updated last year
• KorAP / Tokenizer-Evaluation
  Benchmark scripts for comparing different tokenizers and sentence segmenters of German
  ☆11Updated 2 years ago
• salesforce / simplification
  ☆22Updated 4 months ago
• wjbmattingly / spacyex
  SpaCyEx allows the creation of spaCy Matcher patterns with RegEx like syntax.
  ☆59Updated last year
• nateraw / huggingface-datasets-converter
  Scripts to convert datasets from various sources to Hugging Face Datasets.
  ☆57Updated 2 years ago
• alvarobartt / vertex-ai-huggingface-inference-toolkit
  🤗 HuggingFace Inference Toolkit for Google Cloud Vertex AI (similar to SageMaker's Inference Toolkit, but for Vertex AI and unofficial)
  ☆17Updated last year
• anakin87 / fact-checking-rocks
  Fact checking baseline combining dense retrieval and textual entailment
  ☆29Updated 5 months ago
• raphaelsty / LeNLP
  NLP with Rust for Python 🦀🐍
  ☆62Updated last month
• JulesBelveze / concepcy
  💫 SpaCy wrapper for ConceptNet 💫
  ☆94Updated last year
• masakhane-io / afriqa
  Crosslingual Question Answering for African Languages
  ☆30Updated 8 months ago