稀疏感知低秩表示用于大语言模型高效微调

Adapting large pre-trained language models to downstream tasks frequently necessitates fine-tuning millions of parameters or deploying computationally expensive dense weight updates, severely impeding their deployment in resource-constrained environments. Low-Rank Adaptation (LoRA) mitigates the number of trainable parameters by factorizing weight updates, yet the inherent dense base weights continue to impose substantial storage and computational overheads. Magnitude-based pruning techniques can yield sparse models but typically suffer from performance degradation or require intricate re-training strategies. Addressing these limitations, a novel Sparsity-Aware Low-Rank Representation (SALOR) method is proposed to further enhance fine-tuning efficiency by introducing structured sparsity into the low-rank update matrices. The core principle of SALOR involves leveraging pruning techniques to identify and remove low-contributing elements within the low-rank matrices during the decomposition process, thereby achieving more extreme parameter compression and computational optimization. Specifically, SALOR applies sparsity constraints to the incremental update matrices (ΔW = AB) in LoRA, ensuring that a significant portion of elements in matrices A and B are zero. This approach not only retains LoRA's advantage of reducing trainable parameters but also further decreases storage requirements and inference computational complexity by sparsifying the low-rank decomposition results. Compared to traditional LoRA, SALOR can significantly reduce the number of parameters required for fine-tuning and inference latency while maintaining or even improving model performance. Experimental evaluations demonstrate that SALOR achieves superior parameter and computational efficiency across various downstream tasks compared to LoRA and other pruning methods, positioning it as an ideal solution for deploying large language models on edge devices or in settings with limited computational resources. This technique offers a novel and effective solution for efficient fine-tuning of large language models by integrating low-rank adaptation with structured sparsity.