摩擦的旋转位置嵌入与长输入：几何视角

Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding positions in language models, which, while effective, experiences performance breakdown when input lengths exceed training lengths. Prior analyses have rightly asserted that long inputs cause channels to rotate 'out of distribution,' but the relationship between extra rotation and pathological behavior remains unclear. Through empirical and theoretical analysis, a unified geometric understanding of attention behavior with RoPE is advanced. The study finds that attention induces tight clustering of separated key and query latent point clouds, allowing for the creation of sink tokens: placeholders that enable attention heads to avoid token mixing when not required. The application of RoPE to longer inputs reveals specific geometric characteristics that impact model performance.