Mathematical Application Research Team Meeting #17

Mathematical Application Research Team is delighted to welcome Hugo Koubbi from CEREMADE, Université Paris Dauphine, for an upcoming team meeting. We warmly invite everyone to join us to hear his talk:

Title: Mean-field Transformers

Abstract: Since their introduction, Transformers have achieved unprecedented success across a wide range of deep-learning tasks. In this talk, we present a recent line of work that interprets Transformer architectures as interacting particle systems and studies their continuum, or mean-field, limits. By considering idealized attention dynamics on the sphere, we establish connections between Transformers, Wasserstein gradient flows, and synchronization models such as the Kuramoto model. We describe a global clustering phenomenon in which tokens eventually synchronize, following long-lived metastable regimes characterized by the coexistence of several clusters. Finally, we explain how recent work extends these ideas beyond idealized models and reveals analogous phenomena in more realistic Transformer architectures at initialization.