Neural Weight Norm = Kolmogorov Complexity

Summary

Shows N(s) ≤ K(s) ≤ N(s) log N(s) for the min non-zero parameter count of a fixed-precision looped net that emits s; both sides tight (program-to-weights encoding; permutation-matrix witness). In fixed precision every Lp norm collapses to N(s), so L2 weight decay induces an output prior matching Solomonoff's universal prior up to a log in the exponent. Infinite/rational precision makes the bound vacuous. Conceptual, not small-scale predictive; no experiments. No official code on abs.

Keywords

weight-decay · kolmogorov · solomonoff · looped-transformer · mdl · eth-zurich · fixed-precision

Topics

weight decay, Kolmogorov complexity, Solomonoff induction, looped networks

Research notes

• Primary: arxiv abs (cs.LG / cs.IT). ETH Zürich; correspondence tiberiu@musat.ai. No official code on abs. HF has no paper page (API 404). Discord posted abs. Theory-only; no datasets_local row. ArXiv license widget not visible in converted abs HTML, so paper license left blank.