commoncrawl / web-languagesLinks
Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code
β52Updated this week
Alternatives and similar repositories for web-languages
Users that are interested in web-languages are comparing it to the libraries listed below
Sorting:
- π¬ Language Identification with Support for More Than 2000 Labels -- EMNLP 2023β149Updated 2 months ago
- Libraries, Archives and Museums (LAM)β85Updated 2 years ago
- Efficiently find the best-suited language model (LM) for your NLP taskβ127Updated last month
- A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.β60Updated last year
- Notebooks for training universal 0-shot classifiers on many different tasksβ136Updated 8 months ago
- The robust European language model benchmark.β120Updated this week
- This repository contains an easy and intuitive approach to use SetFit in combination with spaCy.β80Updated 2 years ago
- β109Updated 8 months ago
- Data and evaluation code for the paper WikiNEuRal: Combined Neural and Knowledge-based Silver Data Creation for Multilingual NER (EMNLP 2β¦β69Updated 2 years ago
- Pre-train Static Word Embeddingsβ84Updated 3 months ago
- Generalist and Lightweight Model for Text Classificationβ156Updated 2 months ago
- [EMNLP 2023 Demo] fabricator - annotating and generating datasets with large language models.β110Updated last year
- A BERT-based application for reusable text classification at scaleβ38Updated 2 years ago
- Layout Analysis Dataset with Segmonto (LADaS)β21Updated last month
- Code for SaGe subword tokenizer (EACL 2023)β26Updated 9 months ago
- AfroLID, a powerful neural toolkit for African languages identification which covers 517 African languages.β32Updated 5 months ago
- SpaCyEx allows the creation of spaCy Matcher patterns with RegEx like syntax.β59Updated last year
- Trully flash implementation of DeBERTa disentangled attention mechanism.β63Updated 3 weeks ago
- A library for working with prompt templates locally or on the Hugging Face Hub.β50Updated 5 months ago
- Small python package to measure OCR quality and other related metrics.β25Updated last year
- MAFAND-MTβ57Updated last year
- β67Updated last year
- π’ Work with static vector modelsβ28Updated 4 months ago
- πΊοΈ Data Cleaning and Textual Data Visualization πΊοΈβ185Updated 3 months ago
- The pipeline for the OSCAR corpusβ171Updated last year
- FastFit β‘ When LLMs are Unfit Use FastFit β‘ Fast and Effective Text Classification with Many Classesβ211Updated 3 months ago
- π₯ Use Hugging Face text and token classification pipelines directly in spaCyβ63Updated last year
- A list of awesome open source projects in the machine learning field, who's developers are mainly based in Germanyβ46Updated 11 months ago
- BPE modification that implements removing of the intermediate tokens during tokenizer training.β24Updated 9 months ago
- ParaNames: A multilingual resource for parallel namesβ36Updated last year