Nov 3, 2026 · found Nov 9, 2026, 6:10 AM PT
Collinear and Hugging Face publish CWE-bench v2, 100 defensive patch tasks
Verdict: Vendor-reported · Benchmarks · AI security
A public 100-task benchmark where agents must close security flaws without breaking behavior. The leaderboard numbers come from the benchmark's own authors, and only three of eleven models are named.
A security benchmark asks coding agents to find and fix real weaknesses in open-source code, graded by a deterministic verifier.
- Claim: 100 public tasks across 91 CWE types and eight languages. Gemini 4 Argon leads at 67.8% pass@1 (81.0% pass@4) and Claude Opus 5.5 follows at 67.3% (78.0%). DeepSeek V4.1 Flash scores 57.0% at $0.04 per rollout. Hugging Face
- Evidence: Tasks run in the Harbor harness with four attempts each, and 95 tasks were solved by at least one model. The leader fails 19% of tasks on every attempt. Hugging Face
- Caveats: The post does not say who ran the evaluations, names three of eleven models, and gives no cost method. The first two scores are within half a point. The open leaderboard accepts unverified community results, and the dataset is gated. Dataset card
- Why it matters: Grading on closed exploits and kept behavior is a stricter test of defensive work than finding bugs alone.
Verdict history
- Nov 9, 2026, 6:10 AM PT: Vendor-reported. Blog post and dataset card read. The post does not say who ran the reported evaluations and names only three of eleven models. The dataset is gated, so tasks were not inspected. An explainx.ai summary was used as a lead only. Not checked whether the 71.0% CWE-bench figure in Google's Gemini 4 Carbon post refers to this version.