On the Role of Hidden States of Modern Hopfield Network in Transformer

Large language models such as ChatGPT are based on deep learning architectures known as Transformers. Owing to their remarkable performance and broad applicability, Transformers have become indispensable in modern AI development. However, it still remains an open question why Transformers perform so well and what the essential meaning of their unique structure is. One possible clue lies in the mathematical correspondence between Hopfield Networks and Transformers.

In this talk, I will first introduce the major developments over the past decade that have significantly increased the storage capacity of Hopfield Networks. I will then review the theoretical correspondence between Hopfield Networks and Transformers. Building on this background, I will present our recent findings: by extending this correspondence to include the hidden-state dynamics of Hopfield Networks, we discovered a new class of Transformers that can recursively propagate attention-score information across layers. Furthermore, we found, both theoretically and experimentally, that this new Transformer architecture resolves the “rank collapse” problem often observed in conventional multi-layer attention. As a result, when applied to language generation and image recognition tasks, it achieves performance surpassing that of existing Transformer-based models.