迈向多模态大型语言模型的联邦预训练

The rapid evolution of Multimodal Large Language Models (MLLMs) is hindered by the saturation of high-quality public data, while vast amounts of diverse multimodal data remain locked in privacy-sensitive environments. Federated Learning (FL) presents a promising approach to unlock these distributed resources, yet existing research primarily focuses on fine-tuning, leaving the foundational pre-training phase largely unexplored. This study introduces the Federated MLLM Alignment (Fed-MA) task, a lightweight pre-training paradigm that freezes the vision encoder and the Large Language Model (LLM) while collaboratively training the cross-modal projector. Two critical challenges are identified in this setting: parameter interference during local aggregation and the effective utilization of distributed data resources. This research opens new avenues for pre-training multimodal models.