Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

Summary

Sleep replaces train/test with wake/sleep: Knowledge Seeding distills a smaller self into newly unlocked lower-frequency CMS/MoE experts (on-policy GKD plus RL imitation), then Dreaming generates SEAL-style self-edits with a random extra expert. Nested Learning Hope backbone. Class-incremental CLINC/Banking/DBpedia; long-context RULER/LongHealth/QASPER; CTNL dual-language translation; BABILong; Qwen3-8B AIME-24 79.2 vs OPSD 76.6; SQuAD incorporation above SEAL; ARC few-shot 80% vs SEAL 72.5. No official code on the abs page.

Keywords

sleep · nested-learning · knowledge-seeding · dreaming · continual-learning · google · hope

Topics

continual learning, memory consolidation, nested learning

Research notes

• Primary: arxiv abs (CC BY 4.0, cs.LG). Google / Cornell; correspondence alibehrouz/adeljavanmard/mirrokni@google.com and sh2574@cornell.edu. Discord posted AlphaXiv 2606.03979; canonical abs recorded. HF paper page 29 upvotes, org google (HF author list omits Javanmard; arxiv list used). No official code on abs. Uses existing CLINC/Banking/DBpedia/SQuAD/ARC/RULER; no new standalone corpus, so no datasets_local row.