你的 Nginx 正在扼杀你的 AI 服务——为何需要重新设计流量层

Four key numbers highlight the fundamental challenges in AI infrastructure: the maximum wait time users can tolerate is 3 seconds, beyond which churn increases sharply; the median time for a 70B model to complete a full inference pass on an A100 is 47 seconds; the same model takes 0.3 seconds to output its first token; and the on-demand price for one A100 GPU is $2.48 per hour, which is wasted if it remains idle at 3 AM. The tension between these numbers illustrates the core engineering problem: users demand instant responses, models require time to compute, and resources must be scheduled precisely, yet traditional traffic layers are unaware of these dynamics.