通过基于表示消除的偏好优化对大型语言模型进行去毒化

Large language models (LLMs) trained on web-scale data can produce toxic outputs, raising concerns for their safe deployment. Previous defenses, based on algorithms like DPO and NPO, reduce the likelihood of harmful continuations but are not robust; they are vulnerable to adversarial prompting and can be easily undone by fine-tuning-based relearning attacks. Research indicates that these edits to the model are superficial, as linear probing reveals that harmful 'directions' persist in the representations. To address this issue, Representation Erasure-based Preference Optimization (REPO) is proposed, reformulating detoxification as a token-level preference problem. By employing a novel objective with preference data, the approach aims to eliminate harmful representation directions, thereby enhancing the safety of the model.