📄 中文摘要
大型语言模型（LLM）中的思维链（CoT）推理显著提升了复杂任务的准确性，但由于思考阶段序列过长，导致键值（KV）缓存的内存开销巨大。与传统生成任务中所有令牌同等重要的情况不同，CoT推理更侧重于最终答案，这使得传统的KV压缩策略效果不佳。Crystal-KV是一种旨在解决这一挑战的高效KV缓存管理系统。它引入了“答案优先”（Answer-First）原则，该原则认识到在CoT推理过程中，推理路径中的中间步骤并非都具有相同的长期重要性，而最终答案及其直接相关信息才是长期保留的关键。Crystal-KV通过对KV缓存中的令牌序列进行智能划分，将推理路径分解为多个逻辑段，并根据每个段与最终答案的关联程度赋予不同的保留优先级。具体来说，当模型生成推理步骤时，Crystal-KV会动态评估这些步骤对最终答案的贡献潜力。贡献度较低或冗余的中间推理步骤的KV状态会被积极地压缩或丢弃，而与最终答案高度相关的关键信息则会被优先保留。这种选择性保留机制极大地减少了KV缓存的内存占用，同时确保了模型在生成答案时能够访问到所有必要的上下文信息。Crystal-KV的实现包括一套动态剪枝算法和一种基于答案预测的权重分配机制，用于实时调整缓存中不同部分的重要性。实验结果表明，Crystal-KV在保持甚至提升CoT推理准确性的同时，能够显著降低KV缓存的内存消耗，尤其在处理长推理链时，其效率提升更为明显。该方法为在资源受限环境中部署高效的CoT LLM提供了新的思路。
📄 English Summary
Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle
Chain-of-Thought (CoT) reasoning in large language models (LLMs) substantially enhances accuracy for complex tasks, yet it incurs significant memory overhead due to the extensive think-stage sequences stored in the Key-Value (KV) cache. Unlike conventional generation tasks where all tokens hold uniform importance, CoT places a premium on the final answer, rendering traditional KV compression strategies ineffective. Crystal-KV is an efficient KV cache management system designed to address this challenge. It introduces the “Answer-First” principle, which recognizes that not all intermediate steps in a CoT reasoning path possess equal long-term importance; rather, the final answer and its directly relevant information are critical for long-term retention. Crystal-KV intelligently partitions the token sequences within the KV cache, segmenting the reasoning path into logical chunks, and assigns varying retention priorities based on each segment's relevance to the final answer. Specifically, as the model generates reasoning steps, Crystal-KV dynamically evaluates the potential contribution of these steps to the ultimate answer. KV states corresponding to less contributory or redundant intermediate reasoning steps are aggressively compressed or discarded, while critical information highly pertinent to the final answer is prioritized for retention. This selective retention mechanism drastically reduces KV cache memory footprint while ensuring the model has access to all necessary contextual information during answer generation. Crystal-KV's implementation incorporates a set of dynamic pruning algorithms and an answer-prediction-based weighting mechanism to adjust the importance of different cache parts in real-time. Experimental results demonstrate that Crystal-KV significantly reduces KV cache memory consumption, particularly for long reasoning chains, while either maintaining or improving CoT reasoning accuracy. This approach offers a novel perspective for deploying efficient CoT LLMs in resource-constrained environments.