Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Summary

Systematic layer-wise study of how reinforcement-learning post-training of large language models distributes its gains across transformer layers. Against the usual assumption that every layer should be updated uniformly, the authors find that training a single transformer layer recovers most of the improvement achieved by full-parameter RL training and in some cases surpasses it. They introduce "layer contribution", the fraction of full RL improvement recovered by training one layer in isolation, and measure it across seven models spanning two families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO) and task domains including mathematical reasoning, code generation and agentic decision-making. RL gains concentrate in a small subset of layers, and a consistent structural pattern emerges: high-contribution layers sit in the middle of the stack while layers near the input and output ends contribute substantially less. The resulting layer rankings stay strongly correlated across datasets, tasks, model families and RL algorithms.

Keywords

paper arxiv cs.lg cs.cl reinforcement-learning post-training rlhf grpo layer-wise parameter-efficient llm

Topics

Computer Science

Research notes

• Posted in #random-papers as the PDF link (arxiv.org/pdf/2607.01232); abs page recorded here as canonical. Current version is v2 (2 Jul 2026); subjects cs.LG primary, cs.CL cross-list. Licence on the arXiv record is CC BY-NC-ND. No code or dataset link visible on the abs page. The Discord message already carried a point-up reaction from another member, which is not the curator done-marker.