Statistical Physics of In-Context Learning in Transformer

The pre-trained large model demonstrates the ability to learn from examples, that is, it can infer patterns and generalize from a small number of examples without retraining. How does this ability emerge? This report proposes a physical model mapping of the large model pre-training process, and finds that the training process corresponds to spin condensation, the unique energy ground state will determine the example generalization ability, and the diversity of training data is a key element in algorithm design. This study also reveals that the reasoning process of the large model may be fundamentally different from human thinking.