📄 中文摘要
评估大型语言模型（LLMs）的性能远不止依赖困惑度（perplexity）或BLEU分数等统计指标。对于大多数真实的生成式AI应用场景，理解模型是否比基线模型或早期迭代版本产生更优的输出至关重要。这对于摘要生成、内容创作、问答系统、代码生成等应用尤其关键。当人类评估员数量有限、成本高昂且耗时时，开发一种可扩展、成本效益高且自动化的评估方法变得尤为重要。LLM-as-a-Judge方法通过利用一个功能强大的LLM作为裁判来评估其他LLM的输出质量，从而解决这一挑战。这种方法能够模拟人类评估，对生成文本的多个维度进行评分，例如事实准确性、相关性、流畅性、连贯性、安全性、有用性和简洁性。
📄 English Summary
Evaluating generative AI models with Amazon Nova LLM-as-a-Judge on Amazon SageMaker AI
Evaluating the performance of large language models (LLMs) extends beyond statistical metrics such as perplexity or Bilingual Evaluation Understudy (BLEU) scores. For most real-world generative AI scenarios, it is crucial to understand whether a model is producing better outputs than a baseline or an earlier iteration. This is particularly important for applications like summarization, content generation, question answering, and code generation. When human evaluators are limited in number, costly, and time-consuming, developing a scalable, cost-effective, and automated evaluation method becomes paramount. The LLM-as-a-Judge approach addresses this challenge by leveraging a powerful LLM to act as a judge, assessing the quality of outputs from other LLMs. This method can simulate human evaluation, scoring generated text across multiple dimensions such as factual accuracy, relevance, fluency, coherence, safety, helpfulness, and conciseness. Amazon Nova LLM-as-a-Judge represents an innovation from Amazon in the generative AI space, designed to provide high-quality, scalable, and cost-efficient LLM evaluation. It not only assesses the overall model performance but also delves into the nuances of model outputs, offering more comprehensive insights than traditional metrics. On the Amazon SageMaker AI platform, users can seamlessly integrate Amazon Nova LLM-as-a-Judge to automate their LLM evaluation workflows. This includes setting up evaluation tasks, defining evaluation criteria, running evaluations, and visualizing results. SageMaker provides the necessary tools and infrastructure to support large-scale LLM evaluations, including managing datasets, deploying model endpoints, and handling the storage and analysis of evaluation results.