SWE-bench tests a real issue in an existing repository with tests and dependencies rather than generation of an isolated function. The Verified set is additionally filtered by people, but a score is still not equivalent to engineering work: tests may miss architecture, security and maintainability.
Source: SWE-benchSWE-bench
About this eventThe industry measure: the share of real issues from open repositories closed without a human.