基于偏好约束推断的安全强化学习

Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision-making. However, real-world safety constraints can be complex, subjective, and difficult to specify explicitly. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which are often unrealistic in many real-world applications. The major challenge addressed in this study is how to learn these constraints cheaply and reliably. Inferring constraints from human preferences offers a data-efficient alternative, yet it has been identified that popular Bradley-Terry (BT) models fail to capture the asymmetric and heavy-tailed nature of safety costs, leading to risk underestimation. Understanding this issue remains rare in the literature.