Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

Summary

RL is often credited with improving reasoning at the expense of factual knowledge; this paper instead finds that reasoning models outperform their instruction-tuned versions on factual recall by accessing existing parametric knowledge more effectively. Across five model families, structured prompting that explicitly guides hierarchical traversal recovers most of the gap, suggesting much of the missing knowledge is latent rather than absent. Controlled RL experiments further support this: training on unseen, non-extractable facts improves recall of held-out, frequent but previously inaccessible facts, ruling out simple data exposure. Decomposing the training objective attributes the gain to iterated on-policy exploration. The mechanism appears behaviorally and internally: the reasoning advantage grows with retrieval depth, while layerwise analysis finds similar factual representations but divergent query representations. Distilled models, in contrast, often imitate self-correction without acquiring the exploration needed for navigation. Improving factual recall depends not only on expanding what models know but on teaching them to navigate it, motivating post-training methods that optimize traversal.

Keywords

reinforcement learning · parametric knowledge · factual recall · reasoning · post-training · EMNLP

Topics

reinforcement learning, parametric knowledge, factual recall, reasoning

Research notes

• Discovery: Niloofar Mire (@niloofar_mire, verified; CMU) 2026-09-30: https://x.com/niloofar_mire/status/2105154975475409303
• Paper shared by Alex Zhang (@alexxx_zzz6825) in the quoted thread: https://x.com/alexxx_zzz6825/status/2105148993198047238
• EMNLP oral.
• Submitted 2025-11-08, revised 2026-09-04 (v3), cs.CL.
• Connects to the collection's RL, post-training, knowledge-representation, and interpretability entries.