📄 中文摘要
AI记忆系统的比较通常以检索准确性为排名标准，但却忽视了系统在检索到错误信息时的表现。2026年3月，三篇独立的比较文章评估了AI代理记忆系统，均采用LoCoMo作为基准，且仅以检索命中率进行排名，最终宣称某一系统为优胜者。然而，这些评估没有考虑到在实际应用中更为重要的问题：系统在错误情况下的处理方式。LoCoMo作为基准在检索相关记忆方面表现出色，但并未涵盖系统如何应对错误信息、持有矛盾信念或依赖过时知识的情况。对这些因素的忽视可能导致对AI记忆系统能力的片面理解。
记忆基准测试未能评估的内容
出处: What Memory Benchmarks Don't Test
发布: 2026年3月26日
📄 English Summary
What Memory Benchmarks Don't Test
Comparisons of AI memory systems typically rank based on retrieval accuracy, overlooking how systems behave when they retrieve confidently incorrect information. In March 2026, three independent posts evaluated AI agent memory systems using LoCoMo as a benchmark, ranking them solely by retrieval hit rate and declaring a winner. However, these evaluations failed to address the more critical question for production: how does the system respond when it is wrong? While LoCoMo is an excellent benchmark for assessing the ability to surface relevant memories, it does not account for how systems handle incorrect information, hold contradictory beliefs, or rely on outdated knowledge. Ignoring these factors may lead to a skewed understanding of AI memory system capabilities.