§ feed · storyline

Why we no longer evaluate SWE-bench Verified

Cognition publishes analysis finding SWE-bench Verified contaminated by flawed tests and training leakage, and recommends SWE-bench Pro as a replacement benchmark.

Feb 23 · 12:00:00 · primary fetch1 sourceupdated Feb 23 · 12:00:00

SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress.

Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.

read full article on openai.com ↗

§ sources1 publication · timeline below

openai.comWhy we no longer evaluate SWE-bench Verifiedprimary12:00:00