为何我们不再评估 SWE-bench Verified

SWE-bench Verified is found to be contaminated and mismeasures the progress of frontier coding. OpenAI now recommends SWE-bench Pro, highlighting that flawed tests reward shortcuts and training data leakage inflates scores. Verified may appear effective even when models have not genuinely improved on real tasks. Consequently, it is advised not to rely solely on Verified for model selection or product claims. Instead, running SWE-bench Pro or private holdouts is recommended, and Verified scores should be treated with skepticism. A quick checklist for sane benchmarking includes time-splitting evaluations and conducting duplicate/overlap scans against training data.