用自辩论强化学习为多智能体辩论准备推理语言模型

The reasoning capabilities of large language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative reasoning through Multi-Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales. To bridge this gap, a novel approach called “Self-Debate Reinforcement Learning” (SDRL) is proposed, aiming to enhance LLM reasoning abilities by simulating an internal debate process, thereby better adapting them to multi-agent collaborative environments. The core idea of SDRL is to enable the LLM to play multiple roles during training, generating various reasoning paths and solutions for the same problem, and optimizing its final reasoning outcome through internal evaluation and critique. This is analogous to how humans engage in self-reflection and multi-perspective thinking when facing complex problems. Specifically, SDRL establishes a cyclical feedback mechanism where the LLM generates initial answers and supporting arguments, then generates counter-arguments or alternative solutions, and finally synthesizes this information to arrive at a more robust and accurate conclusion. Each step in this process is evaluated and rewarded, progressively enhancing the LLM's capabilities in debating and synthetic reasoning. By introducing Debate Reward, SDRL encourages the model to generate diverse and persuasive arguments and rebuttals, and rewards its ability to reach higher-quality consensus after synthesizing different viewpoints.