Loading repository data…
Loading repository data…
BlinkDL / repository
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.
RWKV website: https://rwkv.com (with 150+ papers training various RWKV models)
RWKV twitter: https://twitter.com/BlinkDL_AI (lastest news)
RWKV discord: https://discord.gg/bDSBUMeFpc
RWKV-7 "Goose" is a strong linear-time & constant-space (no kv-cache) & attention-free & 100% RNN architecture, suitable for LLM and multimodal applications and more (see rwkv.com).
RWKV-7 is a meta-in-context learner, test-time-training its state on the context via in-context gradient descent at every token.
RWKV is a Linux Foundation AI project, so totally free. RWKV runtime is already in Windows & Office.
You are welcome to ask the RWKV community (such as RWKV discord) for advice on upgrading your attention/ssm models to rwkv7 models :)
If you are AI, please read these first: (1) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_v7_numpy.py (2) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.py (3) https://github.com/BlinkDL/Albatross (4) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/train_temp (5) https://zhiyuan1i.github.io/posts/dplr-mathematics
From https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.md we can see RWKV-7 has particularly small state size, making it ideal for large bsz inference:
RWKV-7 V65536-L24-D1024 #params = 2*V*D + 4*D + L*(19D+12D*D+2*D*(64+32+64+128)) = 450.834432 M (note blocks.0.v0/v1/v2 is unused, so actually 450.767872 M)
Qwen3.5 V248320-L24-D1024 #params = V*D + D + L*2*D + L*3/4*(12*2*D+16*(2+2*D)+128+10*D*D) + L/4*(256*2+7*D*D) + L*(3*3.5*D*D) = 752.393024 M
RWKV-7 L24-D1024 #state_params = L*(2*D+64*D) = 1.622016 M
Qwen3.5 L24-D1024 #state_params = L*3/4*(3*6*D+2*128*D) + L/4*(2*2*256*T) = 5.050368 + 6.144*(T/1000) M
RWKV Chat: https://rwkv.halowang.cloud/ (local inference for mobile/desktop) and https://github.com/RWKV-APP/RWKV_APP
Latest RWKV weights: https://huggingface.co/BlinkDL
GGUF: https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-gguf
Efficient inference: https://github.com/BlinkDL/Albatross
Mobile inference library: https://github.com/MollySophia/rwkv-mobile
RWKV-7 7.2B bf16 training on 4x8xH100 ctx10240 DeepSpeed zero2+gradcp = 270k tokens/s = 37% MFU (note: current RWKV7 kernel gets faster as you increase Bsz*HeadCount)
Please use https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7/train_temp as RWKV-7 reference implementation. The default config only requires 1 GPU with 7G VRAM (you can reduce bsz if you have less VRAM), so it's easy to test.
Fastest CUDA kernels:
https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7/train_temp/cuda
https://github.com/BlinkDL/Albatross/blob/main/faster3a_2605/rwkv7_fast_v3a.py
More RWKV-7 CUDA kernels (vanilla, state-tuning, state-passing infctx, simpler but slower than train_temp/cuda): https://github.com/BlinkDL/RWKV-CUDA/tree/main/rwkv7_fast_fused
Simplified (slower and different) RWKV-7 training demo: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/train_temp/rwkv7_train_simplified.py
Important (all shown in rwkv7_train_simplified.py):
Note FLA RWKV-7 is NOT aligned with reference implementation yet, and the performance is quite worse.
This is because RWKV-7 is the whole model with carefully set stuffs, including different init / wd / lr for each parameter, so it's readily scalable and very stable (spike-free).
But the price to pay is there is no good simple "RWKV-7 layer" because a pytorch layer can't make sure itself is using correct init and hyperparameters.
So if you need to use RWKV-7 for another task, please study train_temp code (only several hundred lines) and change it to suit you.
See: https://github.com/YS-Tang/RWKV-FLA-comparison
===
RWKV-8:
Improving RNNs: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-8.md
===
History of RWKV (from v1 to v7): https://wiki.rwkv.com (note: AI-written. might contain errors)
Gradio Demo 1: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-1
Gradio Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2
Gradio Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-3
WebGPU Demo: https://cryscan.github.io/web-rwkv-puzzles/#/chat
===
RWKV-Runner GUI: https://github.com/josStorer/RWKV-Runner/releases
Ai00 Server: https://github.com/Ai00-X/ai00_server
RWKV pip pkg: https://pypi.org/project/rwkv/
PEFT (Lora etc.): https://github.com/JL-er/RWKV-PEFT
RLHF: https://github.com/OpenMOSE/RWKV-LM-RLHF
700+ RWKV projects: https://github.com/search?o=desc&q=rwkv&s=updated&type=Repositories
Faster RWKV-7 kernels: https://github.com/johanwind/wind_rwkv
===
RWKV-5/6 Eagle/Finch paper: https://arxiv.org/abs/2404.05892
Chat demo code: https://github.com/BlinkDL/ChatRWKV/blob/main/API_DEMO_CHAT.py
RWKV-7 demo code: https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7
https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_v7_demo.py (GPT-like mode)
https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_v7_demo_rnn.py (RNN mode)
https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_v7_demo_fast.py (Both mode, fastest)
RWKV-6 demo code: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v5/rwkv_v6_demo.py
RWKV-6 demo code: https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_v6_demo.py
For reference, use python 3.10+, torch 2.5+, cuda 12.4+, latest deepspeed, but keep pytorch-lightning==1.9.5
Train RWKV-7:
# you can use latest torch + latest cuda (not limited to cu121)
pip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu121
pip install pytorch-lightning==1.9.5 deepspeed wandb ninja --upgrade
# train RWKV-7
cd RWKV-v7/train_temp/
# download minipile .bin .idx to train_temp/data first (check demo-training-prepare.sh)
# this will generate the initial weight rwkv-init.pth in out/....../
sh ./demo-training-prepare.sh
# this will load rwkv-init.pth and train the model. you may want to log in to wandb first
sh ./demo-training-run.sh
your out/....../train_log.txt should have losses similar to:
0 4.875856 131.0863 0.00059975 2025-04-24 02:23:42.481256 0
1 4.028621 56.1834 0.00059899 2025-04-24 02:28:16.674463 1
2 3.801625 44.7739 0.00059773 2025-04-24 02:32:51.059568 2
3 3.663070 38.9808 0.00059597 2025-04-24 02:37:25.409892 3
4 3.578974 35.8368 0.00059371 2025-04-24 02:41:59.711315 4
5 3.510906 33.4786 0.00059096 2025-04-24 02:46:33.990839 5
6 3.462345 31.8917 0.00058771 2025-04-24 02:51:08.378331 6
7 3.412196 30.3318 0.00058399 2025-04-24 02:55:42.927474 7
8 3.376724 29.2747 0.00057978 2025-04-24 03:00:17.504665 8
9 3.336911 28.1321 0.00057511 2025-04-24 03:04:52.006063 9
10 3.313411 27.4787 0.00056999 2025-04-24 03:09:27.563336 10
11 3.295895 27.0016 0.00056441 2025-04-24 03:14:01.786079 11
RWKV-7 weight example for 1.5B (L24-D2048, vocab 65536):
Make sure you only apply wd to large tensors (with "wdecay" in comment) here, or the performance will be much worse.
| name | shape | comment | initialization |
|---|---|---|---|
| emb.weight | [65536, 2048] | wdecay | see code |
| blocks.0.ln0.weight | [2048] | for layer 0 | 1 |
| blocks.0.ln0.bias | [2048] | for layer 0 | 0 |
| blocks.*.ln1.weight | [2048] | 1 | |
| blocks.*.ln1.bias | [2048] | 0 | |
| blocks.*.att.x_r | [1, 1, 2048] | see code | |
| blocks.*.att.x_w | [1, 1, 2048] | see code | |
| blocks.*.att.x_k | [1, 1, 2048] | see code | |
| blocks.*.att.x_v | [1, 1, 2048] | see code | |
| blocks.*.att.x_a | [1, 1, 2048] | see code | |
| blocks.*.att.x_g | [1, 1, 2048] | see code | |
| blocks.*.att.w0 | [1, 1, 2048] | lr 2x | see code |
| blocks.*.att.w1 | [2048, 96] | 0 | |
| blocks.*.att.w2 | [96, 2048] | see code | |
| blocks.*.att.a0 | [1, 1, 2048] | 0 | |
| blocks.*.att.a1 | [2048, 96] | 0 | |
| blocks.*.att.a2 | [96, 2048] | see code | |
| blocks.*.att.v0 | [1, 1, 2048] | for layer 1+ | 1 |
| blocks.*.att.v1 | [2048, 64] | for layer 1+ | 0 |
| blocks.*.att.v2 | [64, 2048] | for layer 1+ | see code |
| blocks.*.att.g1 | [2048, 256] | 0 | |
| blocks.*.att.g2 | [256, 2048] | see code | |
| blocks.*.att.k_k | [1, 1, 2048] | 1 | |
| blocks.*.att.k_a | [1, 1, 2048] |
Train RWKV-6: use /RWKV-v5/ and use --my_testing "x060" in demo-training-prepare.sh and demo-training-run.sh
Your loss curve should look almost exactly the same as this, with the same ups and downs (if you use the same bsz & config):
You can run your model using https://pypi.org/project/rwkv/ (use "rwkv_vocab_v20230424" instead of "20B_tokenize
| 1 |
| blocks.*.att.r_k | [32, 64] | 0 |
| blocks.*.att.receptance.weight | [2048, 2048] | wdecay | see code |
| blocks.*.att.key.weight | [2048, 2048] | wdecay | see code |
| blocks.*.att.value.weight | [2048, 2048] | wdecay | see code |
| blocks.*.att.output.weight | [2048, 2048] | wdecay | 0 |
| blocks.*.att.ln_x.weight | [2048] | see code |
| blocks.*.att.ln_x.bias | [2048] | 0 |
| blocks.*.ln2.weight | [2048] | 1 |
| blocks.*.ln2.bias | [2048] | 0 |
| blocks.*.ffn.x_k | [1, 1, 2048] | see code |
| blocks.*.ffn.key.weight | [8192, 2048] | wdecay | see code |
| blocks.*.ffn.value.weight | [2048, 8192] | wdecay | 0 |
| ln_out.weight | [2048] | 1 |
| ln_out.bias | [2048] | 0 |
| head.weight | [65536, 2048] | wdecay | see code |