低资源环境下领域专用化：大型语言模型中离线响应式知识蒸馏的有效性

Large Language Models (LLMs) demonstrate exceptional performance in general tasks but frequently exhibit hallucinations when confronted with domain-specific or institutional knowledge absent from their pre-training data. Addressing this limitation, an offline response-based knowledge distillation method is introduced to develop high-accuracy specialized assistants under constrained hardware resources. Three distinct data strategies are evaluated: general domain adaptation (15,000 lines), unstructured knowledge injection (500 lines), and structured knowledge injection (500 lines), to ascertain the optimal approach for domain adaptation in low-resource settings. The findings indicate that combining a general LLM with a small volume of target domain data effectively enhances model performance in specific domains, significantly reducing hallucinations and improving knowledge accuracy. Specifically, offline response-based knowledge distillation enables models to learn the response patterns and knowledge of a larger teacher model on domain-specific queries without direct access to the teacher, thereby facilitating the deployment of high-performance specialized models on computationally limited devices. This method not only resolves the limitations of large models in acquiring specific domain knowledge but also provides a viable path for enterprises and institutions to implement customized AI solutions while safeguarding sensitive data.