📄 中文摘要
大型语言模型（LLMs）与人类意图对齐的主要方法是强化学习与人类反馈（RLHF）及其变体。尽管这些方法在实践中表现出色，但其在高维环境下的理论泛化特性仍有待深入研究。为了填补这一空白，本研究在算法稳定性框架下，构建了LLMs在RLHF中线性奖励模型下的泛化理论。具体而言，分析了RLHF算法在训练数据扰动下对模型输出的影响，并以此量化了其泛化误差界限。通过对稳定性的严格数学推导，揭示了模型复杂度、数据量以及奖励模型噪声对泛化性能的关键影响。研究表明，在特定条件下，RLHF能够在高维空间中实现有效的泛化，即使在奖励模型仅为线性近似时也能保持良好的性能。此外，探讨了不同RLHF变体（如PPO、DPO）在稳定性方面的差异，并提供了理论依据来解释它们在实际应用中表现出的不同泛化能力。这些理论发现不仅为理解RLHF的成功提供了新的视角，也为未来设计更具鲁棒性和泛化能力的对齐算法奠定了坚实的基础。通过深入剖析算法的内在机制，为优化LLMs的对齐过程提供了理论指导，有助于开发出更安全、更符合人类价值观的人工智能系统。
📄 English Summary
Towards a Theoretical Understanding to the Generalization of RLHF
Reinforcement Learning from Human Feedback (RLHF) and its variants have become the dominant approaches for aligning Large Language Models (LLMs) with human intent. While empirically effective, the theoretical generalization properties of these methods in high-dimensional settings remain largely unexplored. To address this, a generalization theory for RLHF of LLMs under the linear reward model is developed within the framework of algorithmic stability. Specifically, the impact of training data perturbations on model outputs in RLHF algorithms is analyzed, and generalization error bounds are quantified based on this analysis. Through rigorous mathematical derivations of stability, the critical influences of model complexity, data volume, and reward model noise on generalization performance are revealed. The research demonstrates that, under specific conditions, RLHF can achieve effective generalization in high-dimensional spaces, maintaining robust performance even when the reward model is only a linear approximation. Furthermore, differences in stability among various RLHF variants (e.g., PPO, DPO) are explored, providing theoretical explanations for their observed varying generalization capabilities in practical applications. These theoretical findings not only offer new perspectives for understanding the success of RLHF but also lay a solid foundation for designing more robust and generalizable alignment algorithms in the future. By deeply dissecting the intrinsic mechanisms of these algorithms, theoretical guidance is provided for optimizing the alignment process of LLMs, contributing to the development of safer and more human-aligned artificial intelligence systems.