AMUSE: Anytime Muon with Stable Gradient Evaluation
- Type
- other
- Venue
- arXiv / KAIST / KRAFTON / Seoul National University
Summary
River-valley analysis: Muon orthogonalization enlarges bulk (river) steps but also amplifies dominant-direction noise and valley-wall oscillations. AMUSE interpolates gradient evaluation from the fast Muon sequence toward the averaged sequence via a time-varying coefficient, enabling anytime training without LR schedules. Improves the performance-iteration Pareto frontier over (Schedule-Free) AdamW and Muon on vision tasks and Llama-style FineWeb pretraining (124M/720M/1.3B). Code https://github.com/kjeiun/amuse.
Keywords
amuse · muon · schedule-free · river-valley · kaist · krafton · fineweb · anytime-training
Topics
optimization, Muon, schedule-free, LLM pretraining
Research notes
- Primary: arxiv abs (cs.LG). CC BY 4.0 on HTML. KAIST / KRAFTON / SNU; Kim/Shin equal contrib. Correspondence jueunkim/br.shin/minhaksong/chulhee.yun@kaist.ac.kr, jihuny@krafton.com, bhbaek2001@snu.ac.kr. Code Apache-2.0 https://github.com/kjeiun/amuse (35 stars at check). HF paper page 1 upvote; no linked models/datasets. Discord posted abs plus a note comparing with Aurora. Pretrains on public FineWeb and standard vision sets; no new corpus, so no datasets_local row. License field left blank per catalog convention.