Alternatives and similar repositories for web-languages:

Users that are interested in web-languages are comparing it to the libraries listed below

• r-three / common-pile
  Repo to hold code and track issues for the collection of permissively licensed data
  ☆23Updated 2 weeks ago
• cisnlp / GlotLID
  💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023
  ☆127Updated 5 months ago
• bigscience-workshop / lam
  Libraries, Archives and Museums (LAM)
  ☆82Updated 2 years ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆24Updated 4 months ago
• ltgoslo / ltg-bert
  LTG-Bert
  ☆32Updated last year
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆56Updated 2 weeks ago
• Knowledgator / FlashDeBERTa
  Trully flash implementation of DeBERTa disentangled attention mechanism.
  ☆45Updated 2 weeks ago
• Pleias / marginalia
  ☆67Updated last year
• malteos / llm-datasets
  A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
  ☆58Updated 8 months ago
• cisnlp / Glot500
  Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023
  ☆100Updated last year
• opinionscience / BERTransfer
  A BERT-based application for reusable text classification at scale
  ☆38Updated last year
• chandar-lab / NeoBERT
  ☆46Updated 2 months ago
• huggingface / olm-training
  Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.
  ☆93Updated 2 years ago
• pchizhov / picky_bpe
  BPE modification that implements removing of the intermediate tokens during tokenizer training.
  ☆25Updated 5 months ago
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year
• wjbmattingly / spacyex
  SpaCyEx allows the creation of spaCy Matcher patterns with RegEx like syntax.
  ☆59Updated 11 months ago
• Pleias / Quest-Best-Tokens
  An introduction to LLM Sampling
  ☆77Updated 4 months ago
• bigscience-workshop / metadata
  Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.
  ☆31Updated last year
• oscar-project / ungoliant
  The pipeline for the OSCAR corpus
  ☆168Updated last year
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆101Updated last year
• bltlab / paranames
  ParaNames: A multilingual resource for parallel names
  ☆31Updated 11 months ago
• JulesBelveze / concepcy
  💫 SpaCy wrapper for ConceptNet 💫
  ☆92Updated last year
• explosion / spacy-huggingface-pipelines
  💥 Use Hugging Face text and token classification pipelines directly in spaCy
  ☆63Updated last year
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆21Updated last year
• alvarobartt / vertex-ai-huggingface-inference-toolkit
  🤗 HuggingFace Inference Toolkit for Google Cloud Vertex AI (similar to SageMaker's Inference Toolkit, but for Vertex AI and unofficial)
  ☆17Updated last year
• Knowledgator / GLiClass
  Generalist and Lightweight Model for Text Classification
  ☆123Updated 2 weeks ago
• AnswerDotAI / ModernBERT-Instruct-mini-cookbook
  ☆41Updated 2 months ago
• Babelscape / wikineural
  Data and evaluation code for the paper WikiNEuRal: Combined Neural and Knowledge-based Silver Data Creation for Multilingual NER (EMNLP 2…
  ☆67Updated 2 years ago
• konstantinjdobler / focus
  [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"
  ☆30Updated 6 months ago
• google-research / metricx
  ☆89Updated 4 months ago