稀疏CLIP：对比学习中可解释性与性能的协同优化

Contrastive Language-Image Pre-training (CLIP) has emerged as a cornerstone in vision-language representation learning, underpinning diverse downstream tasks and serving as the default visual backbone in multimodal large language models (MLLMs). Despite its pervasive success, CLIP's dense and opaque latent representations present significant interpretability challenges. A common assumption posits a tension between interpretability and performance: enforcing sparsity traditionally leads to performance degradation. This work introduces a novel approach to co-optimize interpretability and performance in contrastive learning by integrating structured sparsity into CLIP's visual encoder, challenging this conventional wisdom. Specifically, a learnable sparsification mechanism is proposed that dynamically identifies and prunes redundant connections or neurons during training, yielding sparser and more comprehensible visual features without significantly compromising model performance. This sparsity not only contributes to reducing the model's computational complexity and memory footprint but also enhances the transparency of decision-making by reducing feature dimensionality and highlighting salient information. By analyzing these sparse visual features, researchers can gain a deeper understanding of the regions and concepts the model attends to during image-text matching, thereby improving model diagnostics and reliability. Experimental results demonstrate that Sparse CLIP significantly enhances model interpretability while maintaining, and in some cases even surpassing, the performance of the original CLIP across various benchmark datasets, paving a new path towards building more transparent and efficient vision-language models.