Alternatives and similar repositories for wikipedia-downloader:

Users that are interested in wikipedia-downloader are comparing it to the libraries listed below

• EleutherAI / openwebtext2
  ☆87Updated 2 years ago
• EleutherAI / stackexchange-dataset
  Python tools for processing the stackexchange data dumps into a text dataset for Language Models
  ☆80Updated last year
• leogao2 / lm_dataformat
  ☆77Updated last year
• facebookresearch / lss_eval
  This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…
  ☆31Updated last year
• AlanAnsell / peft
  ☆16Updated 6 months ago
• kaiokendev / cutoff-len-is-context-len
  Demonstration that finetuning RoPE model on larger sequences than the pre-trained model adapts the model context limit
  ☆63Updated last year
• EleutherAI / github-downloader
  Script for downloading GitHub.
  ☆90Updated 6 months ago
• CarperAI / squeakily
  A library for squeakily cleaning and filtering language datasets.
  ☆45Updated last year
• jina-ai / textbook
  distill chatGPT coding ability into small model (1b)
  ☆26Updated last year
• neuralwork / arxiver
  Codebase for the arxiver dataset
  ☆13Updated 2 months ago
• EleutherAI / tokengrams
  Efficiently computing & storing token n-grams from large corpora
  ☆17Updated 3 months ago
• huggingface / transformers_bloom_parallel
  Techniques used to run BLOOM at inference in parallel
  ☆37Updated 2 years ago
• leogao2 / commoncrawl_downloader
  ☆32Updated last year
• hpcaitech / Titans
  A collection of models built with ColossalAI
  ☆32Updated 2 years ago
• Agora-Lab-AI / Orca
  An Implementation of "Orca: Progressive Learning from Complex Explanation Traces of GPT-4"
  ☆44Updated 3 months ago
• xingyaoww / LeTI
  Official repo for NAACL 2024 Findings paper "LeTI: Learning to Generate from Textual Interactions."
  ☆63Updated last year
• NolanoOrg / sparse_quant_llms
  SparseGPT + GPTQ Compression of LLMs like LLaMa, OPT, Pythia
  ☆41Updated last year
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆105Updated last month
• LLM360 / Analysis360
  Open Implementations of LLM Analyses
  ☆98Updated 3 months ago
• Lightning-Universe / lightning-ColossalAI
  Large Scale Distributed Model Training strategy with Colossal AI and Lightning AI
  ☆58Updated last year
• young-geng / koala_data_pipeline
  The data processing pipeline for the Koala chatbot language model
  ☆117Updated last year
• hpcaitech / GPT-Demo
  GPT Demo with hybrid distributed training
  ☆10Updated 2 years ago
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆32Updated last year
• cchan / nanoGPT-fp8
  ☆12Updated last year
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆84Updated 3 months ago
• qwopqwop200 / gptqlora
  GPTQLoRA: Efficient Finetuning of Quantized LLMs with GPTQ
  ☆99Updated last year
• chu-tianxiang / QuIP-for-all
  QuIP quantization
  ☆48Updated 10 months ago
• nexusflowai / NexusBench
  Nexusflow function call, tool use, and agent benchmarks.
  ☆19Updated last month
• huggingface / data-measurements-tool
  Developing tools to automatically analyze datasets
  ☆74Updated 3 months ago
• bigcode-project / bigcode-analysis
  Repository for analysis and experiments in the BigCode project.
  ☆117Updated 10 months ago