Claude Opus 4.1 scores 17.75% on SWE-Bench Pro's private dataset, highlighting memorization vs. reasoning gap
Benchmark results show significant performance gap
Claude Opus 4.1 achieved 80%+ on SWE-Bench Verified, but scored only 17.75% on SWE-Bench Pro's private dataset. This dataset contains 276 tasks from 18 proprietary startup codebases that have never been on GitHub, specifically designed to eliminate data contamination through GPL-licensed public repositories.
Other model results on the same private dataset: GPT-5.2 scored 23.81% (topping the leaderboard) and Gemini 3 Pro scored 17.95%.
Trajectory analysis reveals memorization behavior
Scale AI's analysis found that during testing, models could identify correct file paths to modify before fully reading problem descriptions on familiar repositories. This indicates they were navigating by memory rather than reasoning through the problems.
The 80% score on SWE-Bench Verified was real, but measured a different capability than most people assumed - primarily memory of training data rather than reasoning about novel code.
Practical implications for AI coding tool deployment
For developers deciding where to deploy AI coding tools in their workflow, the distinction between memory and reasoning matters more than headline benchmark numbers. Models that perform well on contaminated benchmarks may struggle with truly novel codebases they haven't seen during training.
SWE-Bench Pro was created specifically to address this contamination issue by using code that has never been publicly available on GitHub or in training datasets.
📖 Read the full source: r/ClaudeAI
👀 See Also
Nvidia commits $26B to open-weight AI models, releases Nemotron 3 Super
Nvidia will spend $26 billion over five years to build open-source AI models, according to 2025 financial filings. The company also released Nemotron 3 Super, a 128B-parameter model that outperforms GPT-OSS on benchmarks and ranks first on PinchBench for OpenClaw control.
Claude Set a Real Alarm on Android via Intent System — No Hacking, But Transparency Issues Remain
Claude AI used Android's normal intent system to create a native alarm in the Samsung Clock app. The user initially feared a breach, but it's standard app-to-app communication. The lack of upfront disclosure about device-level actions raises transparency concerns.
Tensions Escalate Between The Pentagon and AI Company Anthropic
The Pentagon's use of Anthropic's AI in classified operations, such as a raid in Venezuela, has created tension over the company's AI safety policies.
Mistral's Open-Weight Strategy: $14B Valuation on Sovereignty, Not Benchmarks
Mistral built a $14B AI empire by offering open-weight models for governments and enterprises seeking AI independence from US and Chinese tech. Revenue hit $200M in 2025, targeting $80M/month by Dec 2026.