一个接一个的偏见：机械奖励塑形与语言奖励模型中的持续偏见

Reward Models (RMs) play a crucial role in the online alignment of language models (LMs) with human preferences. However, preference tuning based on RMs is susceptible to reward hacking, where LM policies learn undesirable behaviors from flawed RMs. By systematically measuring biases in five high-quality RMs, including state-of-the-art models, persistent issues related to length, sycophancy, and overconfidence are identified despite prior work. New biases related to model-specific styles and answer order are also discovered. RM failures are categorized by complexity, and a simple post-hoc intervention is proposed to mitigate low-complexity biases arising from spurious correlations. The proposed mechanistic reward shaping effectively reduces targeted biases without introducing new ones.