Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

Summary

LOTUS places K padded latent blocks between question and answer, loops the backbone R times, and applies per-position CE on gold CoT tokens through the base LM head (LOTUS-aux routes the same targets through a training-only decoder). First latent method reported to bridge explicit CoT at Llama-3.2-3B-Instruct: GSM8K 70.0 vs CoT 71.5 (LOTUS+CODI 70.6); OOD avg 63.9 vs 62.1. Thought-phase 133 vs 338.8 ms (2.5x); natural-language CoT stress 68.13 vs 68.41 with 6.9x thought speedup (140.8 vs 963.6 ms). Post-loop LM-head readout recovers gold CoT (70.9% top-1, 85.8% top-5) and puts mass on unseen-but-valid intermediates. Ablations: looped backbone and parallel gold-CoT CE are both needed; default K=6, c=25, R=6. Chains longer than K fall back to autoregressive completion. Math-only evaluation.

Keywords

lotus · latent-cot · looped-transformers · gsm8k · coconut · sim-cot · microsoft

Topics

latent reasoning, looped transformers, chain-of-thought

Research notes

• Primary: arxiv abs (default nonexclusive-distrib 1.0, cs.LG). Code https://github.com/yingfan-bot/lotus (35 stars at check). Svete ETH Zurich AI Center fellow. Trains on public GSM8k-Aug; no new standalone corpus, so no datasets_local row. HF paper page 2 upvotes. Discord posted abs link.