Spectra：在谱各向异性下重新思考大语言模型的优化器

Gradient signals in training large language models (LLMs) exhibit high anisotropy: recurrent linguistic structures concentrate energy into a small set of dominant spectral directions, while context-specific information resides in a long tail. This spike-tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second moment normalization and tightening the globally stable learning rate bound. Motivated by this analysis, Spectra is proposed as a spike-aware optimizer that suppresses the dominant low-rank spike subspace without amplifying the noise-sensitive spectral tail. Spectra effectively tracks the spike subspace, facilitating improved learning outcomes.