多任务代码大模型：数据混合还是模型合并？

Investigating effective strategies for deploying smaller, specialized code Large Language Models (LLMs) within agentic frameworks, aiming to balance performance, constraints, and costs. This research compares two primary approaches for creating small, multi-task code LLMs: data mixing and model merging. Extensive experiments were conducted across two model families, Qwen Coder and DeepSeek Coder, at two scales: 2B and 7B parameters. The findings indicate that data mixing, by training a single model on multiple tasks, effectively enables the model to learn general representations and shared knowledge across diverse tasks. This approach typically involves carefully designed data sampling and task weighting strategies to ensure robust performance across all target tasks. Conversely, model merging focuses on integrating multiple expert models, each trained for a specific task, through methods such as weight averaging, knowledge distillation, or more sophisticated merging algorithms. This method aims to leverage the specialized knowledge of each expert model to create a versatile model capable of handling multiple tasks. The study meticulously analyzes the performance of both methods across various code-related tasks, including code generation, code completion, and code understanding, evaluating their performance differences, training efficiency, and resource consumption at different model scales. Results reveal the respective advantages and disadvantages of data mixing and model merging in different scenarios. For instance, data mixing might be more suitable for training tasks with high correlation, while model merging could offer better flexibility and performance when task disparities are significant.