Aurora: A Leverage-Aware Optimizer for Rectangular Matrices
- Type
- paper
- Venue
- Tilde Research
- Year
- 2026
- Source
- web
- Access
- free
- Language
- en
- Added
- 2026-08-14T18:56:35Z
- Verified
- 2026-08-14T18:56:35Z
Summary
Tilde Research optimizer (blog 2026-05-05). Muon polar factor inherits row-norm anisotropy on tall matrices; by step 500 more than 1 in 4 SwiGLU MLP neurons can be permanently dead. Aurora is steepest descent under left semi-orthogonality plus uniform row norms: damped alternating row-scale plus Newton-Schulz. Untuned overhead about 6 percent vs Muon. modded-nanoGPT Track 3: Aurora plus Contra-Muon plus u/w-floor reaches val 3.28 in 3175 steps (prior SOTA 3225). 1.1B on about 100B Nemotron-CC tokens: HellaSwag 67.6 percent (+2.5), MMLU +10.8 vs Muon. Code tilde-research/aurora-release; speedrun PR KellerJordan/modded-nanogpt#284.
Keywords
aurora · muon · normuon · optimizer · neuron-death · nanogpt · tilde · leverage · blog · x
Topics
optimizers, Muon, transformer training
Research notes
- Primary: technical blog (cite dewulf2026aurora). Discord/X https://x.com/tilderesearch/status/2052480337591296416 via fxtwitter: Aurora dropping tomorrow. 3175 steps beating NanoGPT Track 3 SOTA by 50 steps. PR https://github.com/KellerJordan/modded-nanogpt/pull/284. Code https://github.com/tilde-research/aurora-release (~152 stars; LICENSE empty at check). Model tilde-research/aurora-1.1B. Optimizer/weights, not a new hosted corpus, so no datasets_local row. Queue item 11 is this same blog; de-dupe later.