Towards Compositional Steepest Descent

Ben Keigwin, Li Yang, Yunzhe Zhang, Dhruv Pai, Alec DeWulf

Type: paper
Venue: Tilde Research
Year: 2026
Source: web
Access: free
Language: en
Added: 2026-08-14T19:06:00Z
Verified: 2026-08-14T19:06:00Z

Summary

Compositional Muon (CM) treats W_Q W_K^T and W_O W_V as the objects of spectral steepest descent, deriving partner-whitened half-split (QK) and hybrid (OV) updates plus an isotropic approximation and a Frobenius gauge correction for momentum. Consistent pretraining gains vs Muon at 340M and 1B; nanoGPT Track 3 PR #311 clears the loss target ~15 steps earlier (2875 vs 2890). Downstream 340M evals mixed. Same lab as Aurora (papers_local 633).

Keywords

compositional-muon · muon · optimizer · qk · ov · nanogpt · tilde · blog

Topics

optimizers, Muon, attention circuits

Research notes

Primary: technical blog (cite keigwin2026compositional). Discord posted the blog. Code https://github.com/tilde-research/comp-muon-release (Apache-2.0, 24 stars / 5 forks at check). Blog byline lists Ashley Zhang among core contributors; bibtex uses Yunzhe Zhang. Correspondence ben@tilderesearch.com, li@tilderesearch.com. Optimizer/code item, not a hosted corpus, so no datasets_local row.