📄 中文摘要
通过在 vLLM 中实现多 LoRA 推理，针对混合专家（MoE）模型进行了优化，展示了内核级别的优化措施。这些优化旨在提高模型推理的效率，尤其是以 GPT-OSS 20B 模型为例，展示了如何在 Amazon SageMaker 和 Amazon Bedrock 平台上高效地服务多个微调模型。该技术为用户提供了更灵活的模型部署和管理方案，能够在不同的应用场景中实现更高的性能和响应速度。
在 Amazon SageMaker AI 和 Amazon Bedrock 上高效服务多个微调模型的 vLLM
📄 English Summary
Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon Bedrock
The implementation of multi-LoRA inference for Mixture of Experts (MoE) models in vLLM showcases kernel-level optimizations aimed at enhancing inference efficiency. Using the GPT-OSS 20B model as a primary example, the work illustrates how to efficiently serve multiple fine-tuned models on Amazon SageMaker and Amazon Bedrock platforms. This approach offers users a more flexible model deployment and management solution, enabling higher performance and responsiveness across various application scenarios.