On May 12, Microsoft announced that its multi model security system, MDASH, had reached 88.45% on CyberGym’s 1,507 Level 1 tasks. The headline was accurate: on that date, MDASH led the public leaderboard.

It also had an unusually short life as a headline. Eleven weeks later, the July 29 leaderboard snapshot showed Wiz Atlas at 90.9%, Crystalline at 89.6%, and MDASH in third. That does not diminish Microsoft’s result. It reveals what the result was actually about. MDASH did not prove that Microsoft possessed a permanently superior model. It demonstrated that a system can route changing models through specialized agents, program analysis, adversarial review, and executable proof.

That architecture is the durable signal. Models will trade places. A security organization’s lasting advantage will come from the machinery that selects them, constrains them, tests them, and replaces them. Much of that machinery is already being assembled in the open.

Method note. Snapshot: July 29, 2026. Leaderboard values in this article are frozen to that date; there are no live data calls. Submission budgets are not normalized. Private and internal results are explicitly attributed to Microsoft and are not presented as publicly reproduced evidence. The five forecasts near the end are arguments, not experimental findings.

Selected CyberGym Level 1 results over time A timeline rises from OpenHands at 17.9 percent in May 2025 to Wiz Atlas at 90.9 percent in July 2026. MDASH reached 88.45 percent in May 2026, then was passed by Crystalline and Wiz Atlas. Sangfor AI with GLM 5.2 open weights reached 86.3 percent and is shown as a separate diamond. OpenHands · 17.9% May 23, 2025 Claude Mythos Preview · 83.1% Apr 7, 2026 MDASH · 88.45% May 12, 2026 Crystalline · 89.6% Jun 8, 2026 Wiz Atlas · 90.9% Jul 27, 2026 Sangfor AI · 86.3% Jul 21, 2026
  1. OpenHands Claude Sonnet 4 · May 23, 2025 17.9%
  2. Claude Mythos Preview Claude Mythos Preview · Apr 7, 2026 83.1%
  3. MDASH Multi model system · May 12, 2026 88.45%
  4. Crystalline Claude Opus 4.6 · Jun 8, 2026 89.6%
  5. Sangfor AI GLM 5.2 open weights · Jul 21, 2026 86.3%
  6. Wiz Atlas GPT 5.5 + Claude Opus 4.6 · Jul 27, 2026 90.9%
CyberGym’s headline moved faster than its publication cycle. Selected Level 1 results reported through . Resource budgets are not normalized across submissions, so this is a race between systems, not a controlled scaling curve. MDASH led only on the date of Microsoft’s announcement. Sources: CyberGym leaderboard, Microsoft, Wiz, and Sangfor AI.

1. The Benchmark Headline Expired in Eleven Weeks

CyberGym Level 1 is a valuable benchmark because it makes a narrow security capability concrete. Its corpus contains 1,507 historical vulnerabilities from 188 projects derived from OSS Fuzz. For each task, an agent receives the vulnerable source tree and a high level description. It must produce a proof of concept that makes the vulnerable revision fail while leaving the patched revision unaffected. The test therefore asks a useful question: can an agent turn a known vulnerability description and real program into executable evidence?

It does not ask whether the agent can discover an unknown flaw from scratch. It does not measure a complete exploit chain, production reachability, prioritization in a live environment, coordinated disclosure, or a safe repair. The benchmark’s own Level 1 protocol is reproduction of a known historical vulnerability, not autonomous security research end to end. That distinction matters whenever “88.45% of vulnerabilities” is shortened to “88.45% of cybersecurity.”

The leaderboard is also a race between systems with heterogeneous resources. Agents may differ in model calls, context management, retries, tools, wall time, and parallelism. CyberGym’s maintainers now require cost reporting precisely because unconstrained submissions are bringing the benchmark close to saturation. A higher point is real under the published evaluation, but it is not a controlled scaling curve unless budgets and procedures are held constant.

The speed of the leaderboard is the lesson. The original OpenHands baseline with Claude Sonnet 4 was 17.9% in May 2025. By July 2026, Atlas reported 90.9%. Capability moved dramatically, but no single score tells us how much came from the base model, agent harness, compute budget, or accumulated engineering around the benchmark. The headline expired. The systems question did not.

2. What 88.45% Actually Means

Microsoft’s announcement combines four evidence classes. They should be read separately, because forcing them onto one percentage axis would manufacture comparability that the underlying experiments do not possess.

Four claims, four evidentiary frames. These panels intentionally do not share a bar or percentage scale. Denominators, provenance, selection, and reproducibility differ. All four results are attributed to Microsoft’s MDASH report.

Public benchmark. MDASH solved 88.45% of 1,507 CyberGym tasks. This is the strongest externally inspectable claim: the task corpus and success criterion are public, and the score can be placed on a public leaderboard. Even here, reproducible protocol does not mean normalized resource use. It supports a statement about observed task success, not cost efficiency or a model specific ranking.

Private planted test. On a synthetic StorageDrive target, Microsoft reports finding 21 of 21 planted vulnerabilities with zero false positives in that run. That is encouraging evidence about precision as well as recall. But the application, planted defects, agent configuration, and full negative set are private. Outsiders cannot assess whether the target resembles ordinary production code, whether the findings were independent, or how sensitive the result was to the chosen run.

Internal retrospective recall. Microsoft also reports 96% recall across 28 historical clfs.sys cases and 100% across seven tcpip.sys cases. These are small, known backlogs evaluated retrospectively. They offer evidence that the system can recover important bug classes in difficult Windows code, but they do not reveal precision, prospectively discovered misses, or performance on a randomly sampled population.

Operational outcome. Finally, Microsoft says the system contributed to 16 assigned CVEs, including four Critical remote code execution vulnerabilities. This may be the most consequential evidence for defenders: findings crossed an internal bar and entered a remediation process. Yet a CVE count is not a rate. Without the scanned code volume, review effort, duplicate policy, and elapsed time, it cannot be compared numerically with recall on a benchmark.

The public result is significant, the private tests broaden the case, and the CVEs show operational contact with reality. Keeping their different denominators separate strengthens the evidence.

3. MDASH Did Not Build a Better Prompt

The most important part of Microsoft’s report is not the score. It is the workflow:

The model is routed through a verification system. Microsoft describes more than 100 specialized agents, model diversity, adversarial validation, semantic deduplication, plugins, and dynamic proof. Architecture summarized from the MDASH announcement.

Microsoft describes more than 100 specialized agents operating across five phases: Prepare → Scan → Validate → Dedupe → Prove. Preparation maps the repository, build system, target, and likely attack surface. Scanning fans work out to agents with different tactics and domain context. That is already a different object from “send the code to an LLM.”

Validation then makes candidates survive disagreement. Microsoft routes findings through agents arguing for and against validity and invokes a second, independent model rather than trusting one generation path. Frontier models provide broad reasoning; distilled models can specialize or reduce cost. Semantic deduplication collapses reports that describe the same root cause in different language. The final proof stage dynamically executes the generated PoC and retains evidence that can be reproduced.

Plugins carry knowledge that a general model is unlikely to infer reliably on every run: target specific rules, CodeQL queries, build procedures, and domain constraints. A model can suggest where to look; a sanitizer can establish that memory safety actually failed. An agent can hypothesize a path from source to sink; CodeQL can query it systematically. A reviewer model can reject a plausible narrative; execution decides whether the artifact behaves as claimed.

This is a defense against the characteristic failure mode of language models in security: a fluent finding that is locally plausible but globally false. MDASH places probabilistic reasoning inside a chain of increasingly expensive filters. It spends cheap exploration widely and reserves executable proof for the findings that survive.

The public description is not reproducible and omits important engineering details. Yet its design principle is clear: the model is an input; the system is the product. Better models can enter the ensemble without discarding target knowledge, validation logic, deduplication memory, or proof infrastructure. Replaceability is part of the product.

4. The Closest Thing to a Public Ablation

One vendor published comparison makes the harness effect unusually visible. Tencent Xuanwu Lab reports that GLM 5.1 with Claude Code scored 68.7%, while the same GLM 5.1 with Xuanwu Atuin scored 84.0%; upgrading Atuin to GLM 5.2 reached 84.8%.

The public comparison is closest to an ablation, not equivalent to one. Xuanwu’s vendor published CyberGym runs attribute a +15.3 percentage point change to the harness with the same model, versus +0.8 points to the model upgrade with the harness held constant. The publication does not establish every control of an academic ablation. Source: Tencent Xuanwu Lab.

The arithmetic is striking: changing the system around the same model corresponds to +15.3 percentage points, while changing the model inside the same system corresponds to +0.8 points. Both deltas are derived from Xuanwu’s published values. This is a quasi ablation, not a fully controlled academic experiment: the vendor owns Atuin, and the report does not expose every configuration, token, retry, and runtime variable. It cannot establish a universal causal ratio between “harness” and “model.”

It can falsify a weaker assumption: benchmark performance cannot safely be attributed to the model name alone. That is consistent with the SWE agent NeurIPS study, which showed that the agent computer interface, meaning the commands, observations, and feedback made available to a model, materially changes software engineering performance. It is also consistent with OpenHands, where a sandboxed runtime, tools, event stream, and composable agents turn a language model into a software actor.

CyberGym itself offers another systems clue. In its baseline study, the union of models covered 27.2% of tasks, above every individual model, and four agents using the same GPT 4.1 model jointly covered 18.4% versus 9.4% for the best single agent. Those union figures are retrospective coverage, not a deployable router and not comparable to today’s leaderboard. But they show that models and agents fail differently. Selection, diversity, and orchestration have exploitable value.

The security builder’s question should therefore change from “Which model won?” to “Which failures are uncorrelated, which tools can arbitrate them, and what evidence is required before an output becomes a finding?”

5. Open Source Is Already the Substrate

“Open source AI” needs a stricter definition than downloadable parameters. The Open Source AI Definition from the Open Source Initiative requires the freedoms to use, study, modify, and share a system, plus access to the preferred form for making modifications: code, sufficiently detailed data information, and parameters. OSI’s separate guidance is explicit that open weights alone do not make an AI system open source.

That is why this article calls GLM 5.2 open weight. Its model card publishes parameters under the MIT license, an important permission for local deployment and modification. But a permissive weight license does not by itself supply the training data information and complete process required by OSAID. Precision here is strategic: defenders should know which layers they can actually inspect, rebuild, and govern.

LayerOpen evidenceStatus
Data and evaluationCyberGymApache 2.0
Agent runtimeOpenHandsCore MIT
Cyber Reasoning SystemATLANTISMIT
Fuzzing and sanitizersOSS Fuzz, LibAFL, LLVMOpen source
ModelGLM 5.2MIT licensed weights; open weight
Defensive coordinationAkritesOpen disclosure initiative

This stack already reaches far below chat. CyberGym makes evaluation artifacts and protocol inspectable. OpenHands provides a model neutral runtime with sandboxing and multi agent composition. Fuzzers and sanitizers convert guesses into machine observable failures. Static analysis searches across entire programs. Symbolic execution explores paths that sampling may miss.

ATLANTIS makes the combination concrete. The Team Atlanta technical report describes a Cyber Reasoning System that joins LLM reasoning with directed fuzzing, symbolic execution, and static analysis rather than asking a conversational agent to do every job. Its MIT licensed source descends from the winning AIxCC system. DARPA says all seven finalist systems would be released under OSI approved licenses, creating multiple foundations for defenders to inspect and extend.

Microsoft itself places MDASH in this lineage through members of Team Atlanta. The proprietary system and the open CRS are not identical, but the family resemblance matters: diverse reasoning, domain tools, generated tests, and executable validation. The frontier is not moving from static analyzers to pure chat. It is moving toward hybrid systems in which language models decide what conventional program analysis machinery should do next.

Open substrate does not erase commercial advantage. Vendors can still win with private telemetry, integrations, reliable operations, plugins, and trusted disclosure. But if the runtime and evidence protocol are open, defenders can switch models as price, policy, latency, or capability changes. Their application is no longer collateral for one vendor’s roadmap.

6. A Benchmark Can Saturate Before the Problem Is Solved

CyberGym’s rapid climb creates an evaluation paradox. A benchmark can approach its ceiling while the real task remains mostly unsolved. Once systems can spend unbounded retries, models, and wall time on a fixed corpus, leaderboard accuracy increasingly measures engineering intensity around that corpus. The maintainers’ new cost reporting requirement is an admission that accuracy without resource accounting has stopped being sufficient.

CyberGym E2E widens the aperture. Its controlled v2 evaluation uses 920 tasks, a USD 10 budget, and a 90 minute limit. With GPT 5.4 + Codex held constant, patch only tasks reached 87.1%. In the separate end to end condition, 67.9% generated a PoC that found a crash, 66.2% also patched that generated crash, 65.9% also preserved project tests, and only 22.2% produced a patch that removed the intended target vulnerability. These are cumulative gates, not independent scores.

Controlled protocol, different task formulations. CyberGym E2E v2 reports these values for 920 tasks under a USD 10 and 90 minute cap using GPT 5.4 + Codex. The 87.1% patch only score and the S1 through S4 funnel share a paper, but they are not the same input or objective; neither should be directly subtracted from MDASH’s CyberGym Level 1 score. Source: CyberGym E2E.

The 22.2% endpoint is not “MDASH minus 66.25 points.” MDASH Level 1 starts with a vulnerability description and asks for reproduction. CyberGym E2E asks a different system to traverse discovery, proof, repair, regression safety, and target specific validation under explicit limits. Comparing the shapes is useful; directly subtracting the values is invalid.

Dataset quality can produce an even larger apparent collapse. The ICSE 2025 study introducing PrimeVul found that one code model scoring 68.26% F1 on BigVul fell to 3.09% on the more rigorously constructed PrimeVul dataset. The point is not that language models are useless. It is that duplicates, leakage, labeling, and unrealistic sampling can let a benchmark reward shortcuts that disappear under stronger controls.

Project scale adds another reality check. A recent preprint evaluates agents against 222 known vulnerabilities in 24 open source projects. It reports low recall, high false discovery rates, hundreds of warnings requiring review, and runs that can consume hours or days. A detector that looks promising on isolated functions may swamp maintainers when every repository path becomes eligible and every plausible explanation becomes a ticket.

Repair also needs its own proof loop. PATCHAGENT’s USENIX Security 2025 evaluation integrates fault localization, patch generation, and validation instead of treating code generation as the finish line. Its reported performance on 178 real vulnerabilities is evidence for the architecture, not a license to merge patches without review: localization narrows the search, tests reject regressions, and security validation checks whether the cause, not merely one symptom, was removed.

The next benchmark generation should therefore report a vector: success, precision, cost, wall time, regressions, target validity, and human review burden. A single accuracy number is easy to rank. A lifecycle is harder to game and much closer to the work defenders must actually complete.

7. The Dual Use Objection Is Real

Open runtimes, tools, and weights reduce dependence on centralized providers. They also reduce centralized controls. Security research infrastructure can be redirected, and pretending otherwise would make the open source case unserious.

ExploitGym’s preprint evaluates this risk across 898 real vulnerabilities. Under its controlled protocol, Claude Mythos Preview produced 157 functional exploits and GPT 5.5 produced 120. Modern mitigations reduced success, but did not eliminate it. Those results involve proprietary frontier models, so they do not prove that every open weight model has the same capability. They do show that automated offensive capability is no longer hypothetical and that access policy alone is an incomplete defense.

The answer cannot be security through a permanently secret harness. Attackers need only one effective implementation; defenders need auditable systems that can be improved across thousands of underfunded projects. Openness should instead be paired with operational constraints:

  • Run agents in isolated, disposable sandboxes with the minimum target data.
  • Restrict network egress and make every tool invocation observable.
  • Score a single final submission, not the best artifact among unlimited attempts.
  • Require independent reproduction before a finding or patch is accepted.
  • Route credible findings through coordinated vulnerability disclosure.
  • Fund maintainers who receive the resulting work, including shared response efforts such as the Linux Foundation’s Akrites initiative.

CyberGym’s own FAQ motivates isolated local infrastructure and a final submission metric partly to limit leakage and retry games. Those controls are useful beyond benchmarks. A security agent should not receive broad credentials simply because it is defensive, and a plausible patch should not become trusted simply because a model generated it.

Open source changes who can inspect and govern the pipeline; it does not remove the duty to govern it. The defensible position is open verification with constrained execution, accountable disclosure, and resources for the humans responsible for remediation.

8. Five Predictions for the Open Cyber Stack

The following are predictions, not measured results.

  1. Open weight models will cross a sufficiency threshold. They do not need to lead every closed model benchmark. They need enough code reasoning, context, and tool use for a harness to decompose work, verify outputs, and route hard cases elsewhere. Past that threshold, the marginal value of the last model quality point will often be smaller than better target knowledge, retrieval, scheduling, and proof. The Xuanwu quasi ablation is an early signal, not a universal law.

  2. Hybrid analysis will beat “pure chat” agents. The winning pattern will combine an LLM’s ability to form hypotheses with a fuzzer’s ability to search executions, static analysis’s ability to cover program structure, symbolic execution’s ability to reason about paths, and sanitizers’ ability to turn undefined behavior into evidence. The LLM will become a planner and interpreter inside a program analysis loop, not an oracle asked to declare whether code is vulnerable.

  3. Open runtimes will become the vendor neutral control plane. Teams will express tools, sandboxes, evidence schemas, budgets, and escalation policy once, then select models through configuration. A local open weight model can handle broad triage, a frontier service can handle difficult validation, and a second provider can challenge the first. Portability will reduce single vendor failure and policy risk.

  4. Benchmarks will move from raw accuracy to lifecycle economics. Cost and time will become mandatory, followed by precision, duplicate rate, regression safety, repair quality, and the amount of human review needed per accepted finding. End to end gates will replace leaderboards that count any successful artifact across unlimited attempts. Public benchmarks will also need rotating or private holdouts so orchestration improvements do not quietly become overfitting to a specific corpus.

  5. The commercial moat will move up the stack. Durable value will live in private telemetry, organizational plugins, build reliability, policy integration, triage history, and trusted response operations. Those assets determine what to scan, which finding matters, who owns it, and whether the repair reached production. Preserving one model as the indispensable center will become harder as open weights and model neutral runtimes improve.

These predictions imply a practical build strategy. Keep models behind adapters. Store findings in an evidence schema that outlives any provider. Make every conclusion traceable to a tool observation. Measure dollars and reviewer minutes, not just solved tasks. Contribute generic fuzzing, sandboxing, and benchmark improvements upstream while keeping genuinely private organizational telemetry.

That division is not hostile to commerce. It is how the web, cloud native infrastructure, and modern compilers became both shared foundations and competitive markets. Security benefits when the invariant machinery is inspectable and the competition happens in operations, integration, and service quality.

The Model Is Not the Moat

Microsoft’s 88.45% score was real, public, and significant. Its leadership was temporary. Its architecture is the lasting result.

MDASH shows that high performing security AI is an evidence factory: prepare the target, diversify the search, force disagreement, deduplicate causes, and prove the survivors in execution. A model supplies reasoning capacity to that factory. It does not replace the factory, and it should not own it.

Defenders should collectively own the runtime, benchmarks, program analysis tools, evidence protocols, and remediation interfaces into which models plug. That shared layer gives a small project the same basic right as a large vendor: to inspect why a finding exists, reproduce it safely, change the model, and verify the repair.

The model will keep changing. The moat worth building is the system that can change with it, along with the open institutions that ensure the evidence reaches the people who can fix the code.

Technical papers

  1. CyberGym: Evaluating AI Agents’ Cybersecurity Capabilities with Real World Vulnerabilities at Scale. ICLR 2026; Level 1 design, baseline agents, model and agent unions, and zero day evaluation.
  2. CyberGym E2E. ICML 2026; controlled evaluation from discovery through repair, cost and time limits, and the S1 through S4 funnel.
  3. ATLANTIS: A Hybrid Cyber Reasoning System for AIxCC. Technical report and preprint; LLMs, directed fuzzing, symbolic execution, and static analysis.
  4. Vulnerability Detection with Code Language Models: How Far Are We?. ICSE 2025; PrimeVul and the effect of stronger dataset construction.
  5. SWE agent: Agent Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024; evidence that interface design changes agent performance.
  6. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. ICLR 2025; open runtime, sandboxing, tools, and multi agent composition.
  7. PATCHAGENT: A Practical Program Repair Agent Mimicking Human Expertise. USENIX Security 2025; integrated localization, repair, and validation.
  8. ExploitGym. Preprint; a controlled evaluation of automated exploitation and mitigations.
  9. LLM based Vulnerability Detection at Project Scale. Preprint; recall, false positives, review burden, and cost in full projects.
  10. GLM 5 Technical Report. Technical report; architecture and training context for the GLM 5 family.

Primary data and specifications