Bi-Mamba: Towards Accurate 1-Bit State Space Models

Shengkun Tang, Liqun Ma, Haonan Li, Mingjie Sun, Zhiqiang Shen

Type: other
Venue: arXiv / MBZUAI / Carnegie Mellon University

Summary

Binarizes Mamba-2 in/out projections (~90%+ of params) with FBI-Linear (sign weights plus per-column α/β) and autoregressive distillation from LLaMA2-7B on Amber. Beats GPTQ-2/3bit and BiLLM post-training binarization on Mamba-2 780M/1.3B/2.7B zero-shot and Wiki2/PTB/C4 PPL; 2.7B avg 50.6 vs Mamba-2 FP 59.6, Wiki2 10.0 vs 9.1. Claims ~5× memory / ~3× throughput vs FP16 2.7B on M4 Pro. Accepted in TMLR 2025. Code https://github.com/Tangshengku/Bi-Mamba.

Keywords

bi-mamba · mamba-2 · 1-bit · quantization · ssm · mbzuai · tmlr · fbi-linear

Topics

quantization, 1-bit SSMs, Mamba

Research notes

Primary: arxiv abs (cs.CL; also cs.AI). License CC BY 4.0 on HTML at check. Comment: Accepted in TMLR 2025. Equal contrib Tang/Ma; corresponding Zhiqiang Shen (zhiqiang.shen@mbzuai.ac.ae). MBZUAI / CMU (Sun). Code https://github.com/Tangshengku/Bi-Mamba (HF githubRepo; 17 stars at check). HF paper page 0 upvotes; no linked models/datasets. Discord posted abs. Trains on public Amber (RefinedWeb/StarCoder/RedPajama), not a new hosted corpus, so no datasets_local row.