Context Language Models

Summary

Introduces Context Language Models (CLMs): language models that natively manage their own context by treating the context as a file and allowing unrestricted updates to it, letting the model learn what is most important to maintain in context and naturally extending to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context-management strategies: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Shifting context management from external harness control to intrinsic model behavior enables both in-context and parametric learning of context-management strategies: steering CLMs with natural-language instructions evolved through a skill-optimization loop improves held-out accuracy by up to 35.9 points while reducing compute. An online RL method for CLMs improves Qwen3.5-9B on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Co-designed Suffix Cache Reuse for CLM serving further cuts server-side compute by 35% relative to standard SGLang at matched performance.

Keywords

agents · long context · context management · RL · multi-agent · inference · caching

Topics

agents, long context, context management, reinforcement learning

Research notes

• Discovery: Aran Komatsuzaki (@arankomatsuzaki) on 2026-09-30: https://x.com/arankomatsuzaki/status/2105181276714242518
• Affiliations: University of Washington, Meta Superintelligence Labs, MIT, Trillium Labs.
• Submitted to arXiv 2026-09-29, cs.AI; CC-BY-4.0.
• Code: https://github.com/facebookresearch/context-language-models
• Connects to the collection's agent, long-context, and RL-for-reasoning entries. discovery tweets https://x.com/natolambert/status/2105284621638439271 (Nathan Lambert) and https://x.com/RulinShao/status/2105282444270448647 (Rulin Shao).