Emergent Analogical Reasoning in Transformers

Gouki Minegishi, Jingyuan Feng, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo

Type: other
Venue: arXiv / The University of Tokyo / Google DeepMind

Summary

Synthetic two-category task with atomic, compositional, and analogical facts. Emergence is sensitive to entity/relation counts, OOD ratio, optimization, and scale. Mechanism: (1) geometric alignment of relational structure in the embedding space (Dirichlet energy) and (2) functor application inside the Transformer. Same qualitative signatures in pretrained Gemma-2-2B/9B and Llama-3.1-8B. ICML 2026 spotlight. Code https://github.com/gouki510/Analogy_in_Transformer.

Keywords

analogy · functor · interpretability · synthetic-task · icml · utokyo · gemma · llama

Topics

interpretability, analogical reasoning, synthetic tasks, category theory

Research notes

Primary: arxiv abs (cs.AI). CC BY 4.0 on HTML; ICML 2026 copyright by the author(s). UTokyo; Furuta at Google DeepMind in an advisory role. Correspondence minegishi@weblab.t.u-tokyo.ac.jp. Code MIT https://github.com/gouki510/Analogy_in_Transformer (34 stars at check). HF has no paper page (API 404). Discord posted abs. Synthetic generator in the repo, not a standalone public corpus, so no datasets_local row. License field left blank per catalog convention.