SoftServe: A Scalable Quasi-Newton Method for Deep Learning

Summary

Quasi-Newton methods are among the most effective for large-scale unconstrained convex optimization, but non-convexity and enormous parameter sizes block their use in deep learning. SoftServe is a family of QN methods that overcome these obstacles without line searches or ad hoc curvature corrections: it derives positive-definite curvature estimates from a variational objective even in the presence of negative curvature, develops diagonal and Kronecker-factored variants that preserve positive definiteness by construction and scale to massive networks, and relies on the stable coupled Newton-Schulz iteration — replacing costly matrix decompositions with GPU-friendly matrix multiplications. Key idea per the announcement: structured curvature approximations (diagonal and Kronecker) plus replacing the exact secant equation with a 'soft' secant penalty. SoftServe excels on severely ill-conditioned problems — recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model — often achieving lower losses than Adam, Muon, and SOAP.

Keywords

SoftServe · quasi-Newton · optimization · secant equation · Kronecker factorization · Newton-Schulz · ill-conditioned · PINN

Topics

quasi-Newton, optimization, curvature, Newton-Schulz, non-convex, deep learning, ill-conditioned

Research notes

• Discovery: @dianarycai (Diana Cai, Cornell) 4-part X thread 2026-10-02 (https://x.com/dianarycai/status/2106020821735256328)
• Co-authors @jooko303, @tanyaisanumber, @gowerrobert; affiliations UMass Amherst, CCM Flatiron Institute, Cornell
• Several ideas draw on the authors' earlier 'Batch and match' BBVI work (arXiv:2402.14758), where score matching in a Gaussian family reduces to solving a quadratic matrix equation; SoftServe updates are likewise based on solutions to quadratic matrix equations
• Paper title stylized 'SOFTSERVE'
• License: CC BY 4.0