📄 中文摘要
外科视觉基础模型（VFMs）在理解外科场景方面展现出强大能力，但现有方法主要依赖单模态RGB预训练，忽视了外科环境中固有的复杂三维几何信息。尽管通用计算机视觉领域有多种架构支持多模态或几何感知输入，但在外科环境中整合深度信息的益处尚未得到充分探索。通过系统性实验，评估了将深度信息纳入外科视觉基础模型预训练的有效性。具体而言，研究了RGB-D预训练对模型在各种外科任务（如器械分割、组织分类和关键点检测）中性能的影响。实验结果表明，在外科场景中，结合深度信息进行预训练可以显著提升模型对三维结构的理解能力，从而提高下游任务的准确性和鲁棒性。尤其是在光照条件不佳或纹理信息有限的复杂手术环境中，深度信息的加入能够有效弥补RGB图像的不足。此外，还探讨了不同深度编码策略（如直接拼接、融合网络或多模态Transformer）对模型性能的影响，并分析了RGB-D预训练对模型泛化能力和数据效率的贡献。研究发现，经过RGB-D预训练的模型在小样本学习场景下表现出更强的适应性，能够更快地收敛并达到更高的性能。这些发现为开发更有效的外科智能系统提供了重要指导，强调了在外科视觉基础模型设计中整合多模态几何信息的必要性。
📄 English Summary
On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training
Vision foundation models (VFMs) have demonstrated significant capabilities in surgical scene understanding. However, current approaches predominantly rely on unimodal RGB pre-training, overlooking the intricate 3D geometry inherent to surgical environments. While various architectures in general computer vision support multimodal or geometry-aware inputs, the specific benefits of incorporating depth information in surgical settings remain largely unexplored. This research conducts a systematic empirical study to evaluate the effectiveness of integrating depth information into the pre-training of surgical VFMs. Specifically, the impact of RGB-D pre-training on model performance across a range of surgical tasks, including instrument segmentation, tissue classification, and keypoint detection, is investigated. Experimental results indicate that incorporating depth information during pre-training significantly enhances the model's understanding of 3D structures in surgical scenes, leading to improved accuracy and robustness in downstream tasks. This advantage is particularly pronounced in complex surgical environments characterized by poor lighting conditions or limited textural information, where depth data effectively compensates for the deficiencies of RGB images. Furthermore, the study explores the influence of different depth encoding strategies, such as direct concatenation, fusion networks, or multimodal Transformers, on model performance. The contributions of RGB-D pre-training to model generalization and data efficiency are also analyzed. Findings reveal that models pre-trained with RGB-D exhibit superior adaptability in few-shot learning scenarios, converging faster and achieving higher performance. These insights provide crucial guidance for developing more effective surgical intelligence systems, underscoring the necessity of integrating multimodal geometric information into the design of surgical vision foundation models.