📄 中文摘要
低秩适应（LoRA）技术在高效微调大型语言模型（LLMs）方面展现出显著优势，仅需引入少量额外参数。然而，现有LoRA方法通常采用静态的秩配置，将相同的秩应用于所有输入token，未能充分考虑不同token之间复杂度和计算需求的变化。针对这一局限，ChunkWise LoRA提出了一种动态且自适应的策略，通过将序列划分为可变长度的块来优化LoRA的应用。这种分块机制根据token的内在特性和计算需求灵活调整，旨在更精细地捕捉序列中不同部分的细微差异。通过这种方式，ChunkWise LoRA能够为每个块动态分配最合适的秩配置，从而避免了统一静态秩配置带来的资源浪费或表达能力不足的问题。
📄 English Summary
ChunkWise LoRA: Adaptive Sequence Partitioning for Memory-Efficient Low-Rank Adaptation and Accelerated LLM Inference
Low-rank adaptation (LoRA) has emerged as a powerful technique for efficiently fine-tuning large language models (LLMs) by introducing minimal additional parameters. However, current LoRA methodologies typically employ static rank configurations uniformly across all input tokens, overlooking the inherent variability in token complexity and computational requirements within a sequence. This work introduces ChunkWise LoRA, a dynamic and adaptive approach designed to address this limitation. ChunkWise LoRA partitions sequences into variable-length chunks, tailoring the LoRA application based on the intrinsic characteristics and computational demands of each segment. This chunking mechanism allows for flexible adjustment of LoRA parameters, enabling the dynamic assignment of optimal rank configurations to individual chunks. By doing so, ChunkWise LoRA circumvents the inefficiencies and potential expressive limitations associated with uniform static rank allocations. The core of this method lies in its intelligent chunking algorithm and adaptive rank assignment strategy, which dynamically adjusts LoRA parameters in real-time according to sequence content to achieve an optimal balance of performance and efficiency. Specifically, ChunkWise LoRA offers two primary advantages: first, it significantly enhances the memory efficiency of LoRA by eliminating the need to maintain high-rank adaptation matrices for all tokens, instead allocating ranks on an as-needed basis according to chunk complexity. Second, through more precise rank allocation, it accelerates LLM inference by reducing unnecessary computational overhead while maintaining or even improving model performance.