扩散引导的语义一致性用于多模态异质性

Federated learning (FL) faces significant challenges due to non-independent and identically distributed (non-IID) client data, which severely impacts global model performance, particularly in multimodal perception scenarios. Conventional approaches often fail to address the semantic discrepancies among clients, resulting in suboptimal performance for multimedia systems that require robust perception capabilities. To tackle this issue, SemanticFL is introduced as a novel framework that harnesses the rich semantic representations from pre-trained diffusion models to provide privacy-preserving guidance for local training. This approach utilizes multi-layer semantic representations from a pre-trained Stable Diffusion model, including VAE-encoded latents and U-Net hierarchical structures, thereby enhancing the model's performance in heterogeneous data environments. SemanticFL effectively improves the accuracy and robustness of multimodal perception tasks.