Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Summary

Four findings: (1) Knowledge flow is asymmetric — language boosts all visual tasks, understanding is a strong prior for generation, generation is largely neutral backward. (2) Task complexity and architecture (shared attention/norm, split FFNs) determine synergy vs competition. (3) Early joint unification beats late alignment; delayed vision causes vision laziness. (4) Asymmetric recipes (e.g. 70/25/5 L/U/G) plus MoE and early unification scale; 13.5B MoE on 2T tokens. Project page https://junlinhan.github.io/projects/physics_of_mm_pretrain/

Keywords

multimodal · unified-pretraining · knowledge-flow · early-fusion · moe · transfusion · meta-fair

Topics

multimodal pretraining, unified models

Research notes

• Primary: arxiv abs. Affiliations FAIR/Reality Labs/Oxford. Correspondence junlinhan@meta.com. Discord posted AlphaXiv 2608.05000.