📄 中文摘要
知识蒸馏为开发强大而高效的小型语言模型（SLMs）提供了一条变革性路径，适用于资源受限的环境。对蒸馏模型与其原始和专有模型的性能及计算成本进行了基准测试，提供了其效率的定量分析。结果表明，蒸馏模型在性能与计算效率之间形成了优越的曲线。研究发现，创建一个蒸馏的8B模型在计算效率上比训练其原始模型高出2000倍，同时在推理能力上与标准模型相当，甚至超过十倍于其规模的模型。这些发现验证了蒸馏不仅仅是压缩技术的有效手段。
蒸馏语言模型的基准测试：资源受限环境中的性能与效率
📄 English Summary
Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings
Knowledge distillation provides a transformative pathway for developing powerful yet efficient small language models (SLMs) suitable for resource-constrained environments. A benchmarking of distilled models against their vanilla and proprietary counterparts was conducted, offering a quantitative analysis of their performance and computational cost. The results demonstrate that distillation yields a superior performance-to-compute curve. It was found that creating a distilled 8B model is over 2,000 times more compute-efficient than training its vanilla counterpart, while achieving reasoning capabilities on par with or even exceeding standard models ten times its size. These findings validate distillation as an effective means of model compression.