微调引导预训练提升下游任务性能

Multimodal pretraining is highly effective for developing general-purpose representations. However, in numerous practical deployments, downstream fine-tuning predominantly relies on a single modality. Conventional pretraining strategies treat all modalities uniformly, which can lead to suboptimal representations for the modality that is critical for the actual application. This paper introduces Finetune-Informed Pretraining (FIP), a model-agnostic methodology designed to bias representation learning towards the primary modality utilized during downstream fine-tuning. The central tenet of FIP involves integrating modality-aware loss functions or attention mechanisms during the pretraining phase. These mechanisms selectively enhance or diminish the contribution of different modalities based on their anticipated importance in downstream tasks. For instance, if a downstream task heavily depends on the text modality, FIP incorporates strategies during pretraining to allocate more resources to learning textual features or assign higher weights to the text modality when integrating multimodal information. This biased learning can be achieved by introducing regularization terms into the pretraining objective function or by adjusting the loss weights for tasks associated with different modalities. Specifically, FIP can leverage prior knowledge about downstream tasks, such as feature importance scores for a particular modality in a downstream task, or estimate modality importance during pretraining using a small subset of downstream task data. By adopting this approach, FIP ensures that the pretrained model generates more robust and discriminative representations for the most crucial modality in downstream tasks, thereby significantly boosting performance on tasks where that modality is dominant.