MrRoPE：混合基数旋转位置编码

Extension of Rotary Position Embedding (RoPE) aims to handle sequences longer than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. MrRoPE (Mixed-radix RoPE) proposes a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies and generalizes prevailing RoPE extension methods. By viewing sequence positions as representations in a specific radix system, MrRoPE extends RoPE's frequency allocation strategy to non-decimal radices. Specifically, it allows for the use of different radices across various dimensions, enabling more flexible and adaptive frequency decay patterns. This mixed-radix approach is better equipped to capture both local and global dependencies in long sequences, as different radices can correspond to distinct periodicities or decay rates. The design of MrRoPE not only encompasses popular methods like linear interpolation and NTK-RoPE as its special cases but also provides a theoretical framework for understanding their underlying mechanisms. This framework, by adjusting radices and associated scaling factors, can flexibly generate new, optimized positional encoding schemes. Experimental results demonstrate that MrRoPE exhibits superior generalization capabilities and performance improvements compared to existing methods when handling long sequence tasks, particularly in mitigating catastrophic forgetting issues when models need to process inputs significantly longer than their pre-training length during inference. Its core idea lies in providing a deeper mathematical interpretation and a broader design space for positional encoding's frequency allocation through radix system conversion, thereby offering new directions for future long-sequence modeling.