动量注意力：上下文学习的物理学与机制可解释性的谱取证

This paper introduces the “Momentum Attention” framework, integrating Transformer's attention mechanism with the physical concept of momentum to deeply understand its in-context learning (ICL) capabilities. We find that during ICL, Transformers simulate particle motion in a potential field via attention, where query, key, and value vectors correspond to particle position, momentum, and energy, respectively. This physical analogy illuminates the intrinsic dynamics of ICL, explaining how models iteratively update internal states to learn and apply new patterns. The paper further introduces “spectral forensics,” a method that quantifies and visualizes model learning behavior across various tasks and data distributions by analyzing the spectral properties of attention matrices. Spectral forensics identifies critical computational patterns and information flow within the model, offering a novel tool for mechanistic interpretability. Experimental results demonstrate that the Momentum Attention framework not only effectively explains Transformer's ICL phenomena but also reveals functional differentiation across different layers and attention heads through spectral analysis. This work provides a physics-based perspective for understanding the emergent capabilities of large language models and lays a foundation for designing more interpretable and efficient AI models in the future.