📄 中文摘要
Pushpay在构建智能体AI解决方案时，面临如何确保生成式AI模型持续高质量输出的挑战。为解决此问题，Pushpay利用Amazon Bedrock创建了一套定制化的生成式AI评估框架。该框架的核心在于自动化和可扩展性，它允许Pushpay对其AI模型的输出进行系统性的质量检查，而不仅仅依赖人工评估。具体而言，该框架集成了一系列评估指标，包括但不限于准确性、相关性、流畅性、安全性以及特定业务场景下的表现。通过定义明确的评估准则和自动化脚本，框架能够定期对AI模型的响应进行采样和分析，从而量化模型的性能。在AWS云环境中，Pushpay能够利用Bedrock提供的强大计算资源和灵活的API接口，快速部署和运行评估任务。评估结果会被汇总并可视化，为开发团队提供清晰的模型性能洞察。更重要的是，这套框架建立了一个快速迭代的反馈循环机制。当评估结果显示模型性能下降或未能满足预期时，开发团队可以迅速识别问题根源，并根据反馈数据对模型进行调优或重新训练。这种持续的质量保证流程不仅提升了AI解决方案的可靠性，也大大缩短了模型改进的周期。Pushpay通过Bedrock的Agentic AI能力，能够将评估过程本身也智能化，例如使用AI辅助生成评估数据集，或利用AI模型对其他AI模型的输出进行初步筛选和评分。这种自举式的评估方法进一步提高了效率和评估的深度。最终，Pushpay成功构建了一个高可靠性、可自我迭代优化的智能体AI系统，为其他企业在生成式AI落地过程中提供了宝贵的实践经验。
📄 English Summary
Build reliable Agentic AI solution with Amazon Bedrock: Learn from Pushpay’s journey on GenAI evaluation
Pushpay's journey in developing a robust Agentic AI solution necessitated a sophisticated approach to ensure continuous high-quality output from its generative AI models. To address this, Pushpay leveraged Amazon Bedrock to architect a custom generative AI evaluation framework. This framework emphasizes automation and scalability, systematically scrutinizing the AI model's outputs beyond mere human review. Specifically, it incorporates a comprehensive set of evaluation metrics, encompassing accuracy, relevance, fluency, safety, and performance within specific business contexts. By defining clear evaluation criteria and employing automated scripts, the framework periodically samples and analyzes AI model responses to quantify performance. Operating within the AWS cloud environment, Pushpay capitalized on Bedrock's potent computational resources and flexible API interfaces for rapid deployment and execution of evaluation tasks. Evaluation results are aggregated and visualized, providing development teams with actionable insights into model performance. Crucially, this framework establishes a rapid iteration feedback loop. When evaluation outcomes indicate a decline in model performance or a failure to meet expectations, development teams can swiftly pinpoint the root cause and proceed with fine-tuning or retraining the model based on the feedback data. This continuous quality assurance process not only enhances the reliability of the AI solution but also significantly accelerates the model improvement cycle. Furthermore, Pushpay utilized Bedrock's Agentic AI capabilities to intelligentize the evaluation process itself, for instance, by using AI to assist in generating evaluation datasets or by employing AI models for preliminary screening and scoring of other AI models' outputs. This self-bootstrapping evaluation methodology further boosted efficiency and the depth of assessment. Ultimately, Pushpay successfully constructed a highly reliable, self-iterating Agentic AI system, offering invaluable practical experience for other enterprises in the deployment of generative AI.