冷启动、模型加载及其对延迟服务水平协议的影响

Cold start latency disrupts Service Level Agreements (SLAs) because 'pod is Running' does not equate to 'model is ready.' In the integration of Kubernetes with vLLM, cold starts encompass image pulls, weight downloads, tensor loading into GPU memory, and warm-up tasks, often related to CUDA graphs. These events are infrequent but have a significant impact, particularly when scaling from zero, dominating the p95/p99 metrics. After deploying a new vLLM revision, the Horizontal Pod Autoscaler (HPA) scales up quickly, and while p50 metrics appear fine, p99 metrics can spike dramatically. Users may be routed to instances that are still undergoing model loading and warm-up, which is not a bug but rather a consequence of physical constraints and orchestration. Running strict SLAs on a GPU fleet highlights the challenges posed by cold start issues.