异质偏好对齐的个性化组相对策略优化

Large Language Models (LLMs) exhibit sophisticated general-purpose capabilities but often struggle to align with diverse individual preferences due to standard post-training methods like Reinforcement Learning with Human Feedback (RLHF), which optimize for a single global objective. Group Relative Policy Optimization (GRPO), a widely used on-policy reinforcement learning framework, implicitly assumes that all samples are exchangeable, inheriting this limitation in personalized contexts. This assumption conflates distinct user reward distributions, systematically biasing learning towards dominant preferences while suppressing minority signals. To address this issue, Personalized Group Relative Policy Optimization (P-GRPO) is introduced, aiming to better handle heterogeneous user preferences and enhance the model's personalization capabilities.