Toward a mechanistic understanding of inference in visual cortex and diffusion models

Summary

Extends sparse coding with a learned pairwise latent interaction matrix M and trains the recurrent ISTA dynamics with denoising score matching plus implicit differentiation. After natural-image training, M recovers collinear V1-like horizontal connections and denoises contours far better than factorial sparse coding, approaching a parameter-matched U-Net. The denoising Jacobian decomposes as Phi J_z Phi^T with sparse-gated lateral spread, explaining contour-tied harmonic bases. On faces, ~30% of latents detach from pixels and act as an emergent hierarchy for global consistency. Yields testable V1 surround-excitation hypotheses.

Keywords

sparse-coding · v1 · diffusion · score-matching · jacobian · contour-integration · neuroscience

Topics

computational neuroscience, sparse coding, diffusion models

Research notes

• Primary: arxiv abs (CC BY 4.0, q-bio.NC / cs.AI). Affiliations UC Berkeley, UC Davis ECE, Flatiron. Correspondence chobitstian@berkeley.edu, baolshausen@berkeley.edu. No code on abs. Discord posted abs link.