Benchmarks

Modern software is increasingly written with AI assistance. This has led to two emerging patterns in professional software development: organizations are producing much more code per developer¹, and as a direct consequence, most codebases now don't exist as a mental model in anyone's head on the team. How do we ensure we're shipping secure code to users?

In a 2025 industry bakeoff of 100+ LLMs, only ~55% of generated code passed basic security checks.² Operational telemetry in another study on real-world repositories over six months showed a 10× rise in new security vulnerabilities in AI-generated changes.³ Asking an LLM to review its own code for security vulnerabilities has not proved helpful either, as LLM-only review exhibited very low recall on critical vulnerability classes across multiple labeled datasets — often zero true positives.⁴

Autofix Bot is purpose-built for securing code. It works in tandem with agentic code generation tools to find and fix security vulnerabilities and hardcoded secrets, with higher accuracy than static-only and LLM-only review, and at a much lower cost than LLM-only review. In this document, we present the architecture, analysis pipeline, performance benchmarks, and future roadmap for Autofix Bot v1.

Hybrid Agent Architecture

We designed Autofix Bot with three key goals:

1. Maximize recall on security vulnerabilities during the AI review of a file while,
2. Keeping false positives low enough to generated remediation patches automatically, and
3. Produce deterministic, reproducible outputs suitable for CI

The agent architecture is hybrid by construction. A semantic/static program analysis provides stable signal; an agentic layer uses that signal (plus code/query tools) to conduct a focused security review, generate remediation patches, and explanation of those patches.

The pipeline contains the following steps:

1. Codebase indexing: Build an AST and whole-project graph (data-flow, control-flow, import graph, sources/sinks) that act as stores. The agent can query these stores during analysis and remediation.
2. Static pass (SAST): Run DeepSource’s SAST to deterministically establish a low-false-positive baseline of known security vulnerabilities. A sub-agent suppresses context-specific false positives.
3. AI review: With static findings, source code tools (ripgrep, graph lookups, etc), and taint analysis sub-agent that tracks the flow of potentially insecure data to inform the main agent's security decisions, the agent performs security review over the relevant slice. The agent also has access to all stores created in step 1, giving it context of the entire codebase and all open-source dependencies.
4. Remediation: Two specialized sub-agents generate fixes for individual issues detected across steps 2 and 3, and explanations where automated fixes are unsafe.
5. Sanitization: A language-specific static harness validates all edits generated in step 4, with an additional AI pass to ensure alignment with the intended fix.
6. Output: Emit a clean git patch, ready to be applied at the HEAD of the branch which was analyzed.
7. Caching: Multi-layered caching for source code, AST, and the project's stores to improve repeat analysis performance.

Empirical Motivation

LLM-only review has well-documented shortcomings that our hybrid architecture directly addresses:

• Low recall on critical CWEs, especially in the presence of other kinds of violations or anti-patterns in the file. A static pass helps steer the LLM review's focus back to security.
• Misses when interprocedural reasoning is required. A grep-only approach lacks deep semantic analysis (can’t track data/control flow across functions), which is a task static analyzers are built for.
• Non-determinism. Re-reviews of the same code produces wildly different results and often misses vulnerabilities flagged in a earlier run. Similar to the first point, a static pass helps steer the output across repeated runs.
• Cost. LLM-only review is expensive, especially for large codebases. Static narrowing trims the search space the agent must read/reason about, reducing prompt/context size and tool invocations.
• Time. Deterministic static seeds lets us shard safely and parallelize analysis without re-reviews, while LLM-only reviewers invariably require additional passes or more tool calls to file search.

Benchmarks: Security Review

For benchmarking the accuracy in finding security vulnerabilities and hotspots, we evaluate Autofix Bot on OWASP/BenchmarkJava⁵ consisting of 2,740 labeled files. We invoke the agent programmatically, processing 15 files in parallel. Autofix Bot attains 88.18% accuracy with True Positive Rate (TPR) 94.35% and False Positive Rate (FPR) 18.43%, completing the suite in ~2.4 hours, costing $58 in token costs.

For comparison, we run Claude Code (Sonnet 4) via its /security-review CLI, with minor prompt adjustments⁶ for file-list input, intentional vulnerability reporting, and JSONL output); OpenAI Codex (gpt-5-medium) using a prompt derived from Claude Code's as Codex CLI doesn't have a security specific review tool yet; and Gemini CLI's Code Review⁷, on the same set of files and sharded 15 files in parallel.

Autofix Bot performs on-par with OpenAI Codex and out-performs Claude Code and Gemini for security review, while being faster and significantly cheaper than all three.

Benchmarks: Secrets Detection

We benchmark three widely used static tools on our proprietary labeled secrets corpus: Gitleaks⁸, Detect-Secrets⁹, and TruffleHog¹⁰; and compare the results with Autofix Bot's secrets detection sub-agent on the same corpus.

We observe that Gitleaks and TruffleHog prioritize precision (0.97 and 0.83) but miss many true secrets (recall 0.62 and 0.27). In contrast, detect-secrets achieves similar recall to Gitleaks (0.61) but with substantially lower precision (0.68), resulting in more false positives (152) than Gitleaks (10) or TruffleHog (29).

Static-only signal is necessary but insufficient. To address the above failure modes, Autofix Bot's secrets detection sub-agent runs:

1. a static regex/pattern sweep to over-approximate candidates (maximizing recall), then
2. a custom fine-tuned classifier to confirm/deny each candidate and extract the value

On this corpus, Autofix Bot achieves the highest F1 (0.9278), substantially ahead of static scanners (Gitleaks 0.7562, detect-secrets 0.6409, TruffleHog 0.4122).

We choose F1 as the headline metric here because secrets datasets are imbalanced and teams care about two things at once: not missing real leaks (recall) and not drowning reviewers in noise (precision). By maximizing F1 — rather than accuracy, which can be inflated by abundant true negatives — Autofix Bot demonstrates the best balance of safety and triage cost. Practically, this means the system catches more real secrets while keeping alerts actionable, making it suitable for always-on CI and large repo backfills.

Benchmarks: Remediation

On OWASP/BenchmarkJava, we evaluate end-to-end fix quality using a shared LLM judge (GPT-4o) with a common prompt. Autofix Bot produced 1,293 correct fixes out of 1,354 (95.49%), with $8.33 total spend and 12.71s average time per fix. Codex (gpt-5 medium) achieved the highest fix accuracy (97.09%) but at $45 and 53.04s average latency; Claude Code was comparable in accuracy (95.48%) at ~$91 and 35.57s per fix.

Autofix Bot delivers near-parity fix accuracy while being ≈4.2× faster than Codex (and ≈2.8× faster than Claude) and ≈5x to 10× cheaper overall.

What's Next

Autofix Bot is currently in early access and we're testing the pre-release with trusted partners. Over the next few weeks, we will be releasing the REST API and the Terminal UI, shortly followed by the GitHub pull request integration. We'll continue to work on our agent architecture and expect more improvements in the metrics we've showcased in this document.

Please follow @autofixbot for updates, and keep an eye on the News feed.