Blealtan / RWKV-LM-LoRA
RWKV is a RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
☆404Updated last year
Related projects: ⓘ
- A torchless, c++ rwkv implementation using 8bit quantization, written in cuda/hip/vulkan for maximum compatibility and minimum dependenci…☆304Updated 7 months ago
- ☆80Updated 4 months ago
- ☆533Updated 9 months ago
- This project is established for real-time training of the RWKV model.☆50Updated 4 months ago
- RWKV infctx trainer, for training arbitary context sizes, to 10k and beyond!☆131Updated last month
- ☆453Updated 11 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆127Updated 2 months ago
- ☆72Updated 2 weeks ago
- INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model☆1,403Updated last month
- Train llama with lora on one 4090 and merge weight of lora to work as stanford alpaca.☆50Updated last year
- ChatGPT-like Web UI for RWKVstic☆100Updated last year
- 📖 — Notebooks related to RWKV☆59Updated last year
- Official repository for LongChat and LongEval☆505Updated 3 months ago
- Implementation of the RWKV language model in pure WebGPU/Rust.☆227Updated last month
- Enhancing LangChain prompts to work better with RWKV models☆34Updated last year
- A localized open-source AI server that is better than ChatGPT.☆456Updated last week
- 4 bits quantization of LLaMa using GPTQ☆129Updated last year
- Simple, hackable and fast implementation for training/finetuning medium-sized LLaMA-based models☆143Updated this week
- Framework agnostic python runtime for RWKV models☆144Updated last year
- Tune any FALCON in 4-bit☆469Updated last year
- A QQ Chatbot based on RWKV (W.I.P.)☆78Updated 10 months ago
- ☆119Updated 9 months ago
- A full pipeline to finetune Vicuna LLM with LoRA and RLHF on consumer hardware. Implementation of RLHF (Reinforcement Learning with Human…☆206Updated 4 months ago
- Xwin-LM: Powerful, Stable, and Reproducible LLM Alignment☆1,016Updated 3 months ago
- Landmark Attention: Random-Access Infinite Context Length for Transformers☆405Updated 9 months ago
- Windows compile of bitsandbytes for use in text-generation-webui.☆341Updated 10 months ago
- Finetuning Large Language Models on One Consumer GPU in Under 4 Bits☆697Updated 3 months ago
- 使用Gradio制作的基于RWKV的角色扮演的webui☆220Updated 3 weeks ago
- Merge Transformers language models by use of gradient parameters.☆193Updated last month
- ☆520Updated 8 months ago