📄 中文摘要
在修改提示、替换模型或调整工具后，工程师们常常面临一个简单却重要的问题：代理的整体表现是变好还是变差？使用聚合指标（如平均成功率和总令牌数）虽然看似正常，但具体任务类型可能会出现隐性故障。简单任务的改善可能掩盖了困难任务的回归，导致问题在生产环境中被忽视。尝试过的几种方法包括使用 LLM 作为评判者进行评分，但由于结果不一致，难以判断分数变化是否真实；手动抽查虽然可以发现问题，但效率低下且难以全面覆盖。最终，找到了一种有效的方法来评估代理的表现。
如何在发布更改之前测试你的 LLM 代理？
出处: How do you test your LLM agents before shipping changes?
发布: 2026年3月24日
📄 English Summary
How do you test your LLM agents before shipping changes?
Engineers often face a crucial question after modifying prompts, swapping models, or tweaking tools: has the agent's overall performance improved or worsened? While aggregate metrics like average success rates and total tokens may appear fine, specific task types can silently fail. Improvements in easier tasks can mask regressions in harder ones, leading to unnoticed issues in production. Various methods have been attempted, including LLM-as-judge scoring, which proved inconsistent and made it difficult to determine if score changes were genuine or just statistical noise. Manual spot-checking, while useful, is inefficient and cannot cover all cases. Ultimately, a reliable method for evaluating agent performance was discovered.