bigcode-project / bigcode-tokenizer

Related projects: ⓘ

• argilla-io / distilabel-spin-dibt
  Repository containing the SPIN experiments on the DIBT 10k ranked prompts
  ☆22Updated 6 months ago
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆32Updated last year
• sgugger / torchdynamo-tests
  ☆20Updated last year
• pacman100 / peft-codegen-25
  ☆24Updated last year
• nbroad1881 / strideformer
  Using short models to classify long texts
  ☆20Updated last year
• CarperAI / squeakily
  A library for squeakily cleaning and filtering language datasets.
  ☆45Updated last year
• conceptofmind / SemDeDup
  ☆18Updated this week
• IlyasMoutawwakil / py-txi
  A Python wrapper around HuggingFace's TGI (text-generation-inference) and TEI (text-embedding-inference) servers.
  ☆31Updated 2 weeks ago
• huggingface / olm-training
  Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.
  ☆91Updated last year
• Alignment-Lab-AI / datagen
  a pipeline for using api calls to agnostically convert unstructured data into structured training data
  ☆26Updated last year
• EleutherAI / lm-eval2
  ☆13Updated this week
• LAION-AI / Anh
  Anh - LAION's multilingual assistant datasets and models
  ☆27Updated last year
• gonglinyuan / metro_t0
  Code repo for "Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers" (ACL 2023)
  ☆22Updated 10 months ago
• pchizhov / picky_bpe
  BPE modification that implements removing of the intermediate tokens during tokenizer training.
  ☆13Updated last week
• deep-diver / LLM-Pref-Mark-UI
  ☆37Updated last year
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆82Updated 6 months ago
• ChrisHayduk / qlora-multi-gpu
  QLoRA with Enhanced Multi GPU Support
  ☆36Updated last year
• google-research-datasets / QAmeleon
  QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…
  ☆33Updated last year
• Upaya07 / NeurIPS-llm-efficiency-challenge
  Code for NeurIPS LLM Efficiency Challenge
  ☆52Updated 5 months ago
• malteos / clp-transfer
  Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning
  ☆29Updated last year
• huggingface / transformers_bloom_parallel
  Techniques used to run BLOOM at inference in parallel
  ☆37Updated last year
• r-three / RAD
  Reference implementation for Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
  ☆40Updated 8 months ago
• facebookresearch / lss_eval
  This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…
  ☆31Updated last year
• EleutherAI / semantic-memorization
  ☆44Updated 2 months ago
• CarperAI / decontamination
  This repository contains code for cleaning your training data of benchmark data to help combat data snooping.
  ☆25Updated last year
• ConiferLabsWA / flan-ul2-alpaca
  ☆32Updated last year
• nateraw / spaces-docker-templates
  🚀🤗 A collection of templates for Hugging Face Spaces
  ☆35Updated 11 months ago
• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆109Updated 2 months ago
• nateraw / huggingface-datasets-converter
  Scripts to convert datasets from various sources to Hugging Face Datasets.
  ☆57Updated last year
• ContextualAI / CLAIR_and_APO
  Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  ☆39Updated 3 weeks ago