📄 中文摘要
大型语言模型（LLM）驱动的多智能体系统正在深刻改变企业自动化领域，然而，评估其工具使用可靠性的系统性方法仍处于早期发展阶段。开发了一个全面的诊断框架，该框架利用大数据分析技术，专注于评估智能体系统中的过程可靠性。该框架旨在满足隐私敏感环境中中小型企业（SME）部署的关键需求。通过对智能体在复杂任务执行过程中调用外部工具的准确性、及时性和一致性进行量化分析，揭示了导致工具调用失败的常见模式。具体来说，框架通过追踪智能体决策路径、分析工具接口参数匹配度以及监控工具执行结果，识别出如语义理解偏差、上下文漂移、API调用格式错误以及工具响应解析失败等问题。此外，该框架还引入了一套度量指标，包括工具调用成功率、失败重试次数、任务完成时间以及资源消耗，以提供多维度的可靠性评估。针对在隐私敏感场景中的应用，框架设计了去识别化和数据最小化策略，确保在收集和分析智能体行为数据时，最大程度地保护用户隐私。通过这些诊断结果，企业能够识别并纠正智能体设计中的缺陷，优化工具集成策略，从而提升多智能体系统在实际业务场景中的稳定性和效率。此方法为构建更健壮、更可信赖的LLM驱动自动化解决方案提供了坚实基础。
📄 English Summary
When Agents Fail to Act: A Diagnostic Framework for Tool Invocation Reliability in Multi-Agent LLM Systems
Multi-agent systems powered by large language models (LLMs) are profoundly transforming enterprise automation, yet systematic evaluation methodologies for assessing tool-use reliability remain underdeveloped. A comprehensive diagnostic framework is introduced, leveraging big data analytics to evaluate procedural reliability in intelligent agent systems. This framework addresses critical needs for SME-centric deployment in privacy-sensitive environments. By quantitatively analyzing the accuracy, timeliness, and consistency of external tool invocations during complex task execution by agents, the framework uncovers common patterns leading to tool invocation failures. Specifically, it identifies issues such as semantic understanding biases, context drift, API call format errors, and tool response parsing failures by tracing agent decision paths, analyzing tool interface parameter matching, and monitoring tool execution results. Furthermore, the framework introduces a suite of metrics, including tool invocation success rate, number of retry attempts upon failure, task completion time, and resource consumption, to provide a multi-dimensional reliability assessment. For applications in privacy-sensitive scenarios, the framework incorporates de-identification and data minimization strategies, ensuring maximum protection of user privacy during the collection and analysis of agent behavior data. These diagnostic results enable enterprises to identify and rectify flaws in agent design, optimize tool integration strategies, thereby enhancing the stability and efficiency of multi-agent systems in real-world business contexts. This approach provides a solid foundation for building more robust and trustworthy LLM-driven automation solutions.