Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland, Jiaju Lin, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Cahya Wirawan, Stanisław Woźniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, Rui-Jie Zhu

Open source

Type: other
Venue: arXiv / RWKV Project / EleutherAI

Summary

Eagle adds matrix-valued WKV states, head LayerNorm, SiLU gating, and drops sigmoid receptance vs RWKV-4. Finch adds data-dependent token-shift and decay via LoRA-style offsets. New RWKV World Tokenizer (65,536, trie greedy) and World v2 mix (~1.12T tokens, ~70/15/15 EN/multilingual/code). Eagle 0.46–7.5B and Finch 1.6/3.1B Apache-2.0 on HF. Eagle-7B multilingual avg 58.2 vs Mistral-7B 55.5 / Llama-2-7B 54.3; English avg 71.5 vs Mistral 75.8. Finch ~4.2× faster than FlashAttention-v2 at seq 16k. Code https://github.com/RWKV/RWKV-LM; models https://huggingface.co/RWKV.

Keywords

rwkv · eagle · finch · linear-attention · rnn · multilingual · world-v2 · eleutherai

Topics

RNN language models, linear attention, RWKV

Research notes

Primary: arxiv abs (cs.CL; also cs.AI). License CC BY 4.0 on HTML at check. Equal first Peng/Goldstein; others alphabetical. RWKV Project (Linux Foundation AI & Data) / EleutherAI / Recursal / many universities. Models Apache-2.0 at https://huggingface.co/RWKV (e.g. RWKV/v5-Eagle-7B-HF 7,053 dl / 72 likes at check) and BlinkDL. Code https://github.com/RWKV/RWKV-LM; inference https://github.com/RWKV/ChatRWKV; time-parallel https://github.com/RWKV/RWKV-infctx-trainer (HF githubRepo; 149 stars at check). HF paper page 39 upvotes; official-looking linked models BlinkDL/rwkv-6-world etc. not copied into hf_* fields. Discord posted abs. World v2 is described as a public 1.12T mix of existing sources; no single hosted v2 card found at check (later Goose-World/RWKV-World-v3 exists), so no datasets_local row.