How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

Summary

Measures the value of one recurrence in a looped (depth-recurrent) transformer in equivalent unique parameters. From an iso-depth pretraining sweep over recurrence counts r in {1,2,4,8} spanning ~50x in training compute, the authors fit a joint scaling law L = E + A(N_once + r^phi N_rec)^(-alpha) + B D^(-beta) and estimate a recurrence-equivalence exponent phi = 0.46 — between full equivalence (phi=1, looping a block r times equals r unique blocks) and no capacity gain (phi=0).

Keywords

looped transformers · scaling laws · recurrence · architecture

Topics

looped transformers, scaling laws, recurrence, architecture

Research notes

• Method: Iso-depth pretraining sweep across recurrence counts spanning ~50x training compute; joint scaling-law fit with a recurrence-equivalence exponent; case studies on truncated backpropagation and hyperconnections as diagnostics.
• Key findings: Recurrence-equivalence exponent phi = 0.46: replacing unique blocks with shared recurrences increases validation loss at matched training compute (e.g., at r=4 a 410M looped model matches a 580M non-looped model but costs as much to train as a 1B non-looped one); Commonly used truncated backpropagation lowers phi to 0.38 — the loop mechanism is poorly trained under truncation even though validation loss decreases; Hyperconnections raise phi to 0.65, a genuine capacity gain; phi separates true loop improvements from training-side gains, which raw validation loss cannot
• Limitations: Iso-depth sweep design; results are for the specific looped-transformer family studied and may not transfer to other recurrent architectures.
• Original Discord link was the /pdf/ form.