IntSeqBERT：通过模谱嵌入学习OEIS中的算术结构

The research presents IntSeqBERT, a dual-stream Transformer encoder designed for masked integer-sequence modeling on the OEIS. Integer sequences in the OEIS range from single-digit constants to astronomical factorials and exponentials, posing challenges for standard tokenized models that cannot handle out-of-vocabulary values or exploit periodic arithmetic structures. Each sequence element is encoded along two complementary axes: a continuous log-scale magnitude embedding and sin/cos modulo embeddings for 100 residues (moduli 2 to 101), fused via FiLM. Three prediction heads—magnitude regression, sign classification, and modulo prediction for 100 moduli—are jointly trained on 274,705 OEIS sequences. At a large scale (91.5M parameters), IntSeqBERT demonstrates its effectiveness in sequence prediction.