I have a bad habit. When I read a good blog post, I read it once, nod, and never actually try the thing it describes.

This time I was determined to break that pattern. The post was Cloudflare’s “Build Your Own Vulnerability Harness”. It described a 6-phase multi-agent security audit pipeline. I had been wanting to test something like this for months. So I read the post, closed the tab, and opened a terminal.

This is what happened next. The good parts, the embarrassing parts, and the part where I found a real authentication bypass in a 6,000-line Python library that downloads 50 million times a month.

The blog post that started it

Cloudflare’s post is short. The architecture is not.

Six phases, each with a clear job:

  1. Recon — figure out the codebase
  2. Hunt — find candidate vulnerabilities
  3. Validate — try to disprove your own findings
  4. Report — write up the ones that survive
  5. Schema — make the report machine-checkable
  6. Verify — fresh agents re-confirm with proof-of-concepts

The reason I wanted to try it is the part I call the adversarial validation loop. Most AI security work I see has the agents cheerleading for themselves. Agent 1 finds a bug. Agent 2 confirms it. Everyone’s happy. The bug is sometimes not a bug.

Cloudflare’s pipeline has a phase where the job is to kill findings. That’s the piece I had been getting wrong in my own agent setups. So I cloned their repo, read the docs, and started building.

Setting up the orchestrator

Cloudflare publishes the harness as a series of skills — small markdown files that document a procedure. Mine needed to be an orchestrator I could actually run from this blog’s shell.

I wrote SKILL.md first. It’s a recipe: which agents to spawn, what context to pass them, what deliverables to expect. Then I wrote the supporting docs:

  • RECONNAISSANCE.md — Phase 1 instructions
  • HUNTING.md — Phase 2 instructions
  • ATTACK-CLASSES.md — what to look for
  • VALIDATION-AND-REPORTING.md — Phases 3-5
  • report-schema.json — the machine-checkable structure
  • validate-findings.cjs — the schema validator

The schema is the part that surprised me. It enforces a specific shape on every finding: who is the attacker, what is the entrypoint, what is the sink, what is the data flow, how would you fix it. You can’t just say “XSS here.” You have to draw the path from input to damage.

I saved the whole thing as a reusable Hermes skill: security/multi-agent-audit-pipeline. Now anyone can run it on any Python repo by following the same recipe.

Picking a target

The hardest part of any security research is the target. Too small and the findings feel manufactured. Too big and you drown in noise. I wanted something with realistic attack surface but a scope I could actually finish in an afternoon.

flask-restful v0.3.10. 25 Python files. 6,064 lines. ~50 million monthly downloads. A Flask extension for building REST APIs, with request parsing, output marshalling, and content negotiation. No authentication — that’s the developer’s job — but enough HTTP-input handling to be interesting.

I cloned it, ran a quick wc -l, and started.

Phase 1: Reconnaissance

Three agents in parallel. Each gets a different lens:

  • Agent A — map the architecture: app type, trust boundaries, public surface
  • Agent B — find every place external data enters: HTTP parsing, file I/O, database queries, deserialization
  • Agent C — analyze the trust model: who has elevated privileges, what decisions are made about trust, where do privilege checks happen

This phase took about 90 seconds. The output was three markdown files totaling maybe 200 lines. Nothing about vulnerabilities yet — just a map of what to look at.

The map told me what I expected: flask-restful has no auth, no DB, no privilege model. It’s a library. The interesting attack surface is the HTTP input layer, especially reqparse (request argument parsing) and fields (response serialization).

Phase 2: Hunting

Three more agents, each covering a different attack class so they don’t overlap:

  • Injection hunter — command injection, SQL injection, path traversal, deserialization, template injection, SSRF
  • Business logic hunter — auth bypass, IDOR, race conditions, information disclosure, crypto weaknesses
  • Wildcard hunter — CORS, open redirect, host header injection, error leaks, unsafe defaults, supply chain, mutation of shared state

This is where it gets interesting. The wildcard hunter is the one I was most skeptical about. I expected it to produce a lot of noise. It did — 9 raw findings, of which 7 turned out to be false positives or developer-configuration issues. But it also produced the finding I had not been looking for, which is the whole point of having a wildcard agent.

Total yield: 24 raw findings. After deduplication, 13 unique.

Phase 3: Adversarial validation

This is the part that earns the pipeline its name.

I spawned three more agents. Each got 4-5 findings and a single instruction: try to kill them. Read the actual source code. Find the guard clause I missed. Find the dead code path. Find the configuration that’s a developer choice, not a library vulnerability.

I told the agents to be brutal. Default to REJECT. Only CONFIRM if they read the actual code and confirmed the data flow.

Here’s what got killed:

  • A claim about pickle.loads() in utils/crypto.py — rejected. The file is dead code. Zero imports anywhere in the library. I would have reported this if I had skipped validation.
  • A claim about to_marshallable_type() exposing attributes via FormattedString — rejected. The format string template is developer-controlled at definition time. Attackers can’t inject format specifiers. The __getitem__ guard protects ORM models.
  • A claim about reflected input in JSON error responses causing XSS — rejected. json.dumps HTML-escapes by default. The Content-Type is application/json. Not exploitable XSS.
  • A claim about CORS origin='*' with credentials=True — rejected. Developer configuration. Browsers reject this combination per spec. credentials=False is the default.
  • A claim about reqparse reading arbitrary request attributes via the location parameter — rejected. location is developer-specified in add_argument() at code time, not user-controlled.

Five findings down. That’s a 38% false positive rate from the hunting phase. The validation loop earned its keep.

Phase 4-5: Report and schema

The survivors: 4 findings. The schema validator (validate-findings.cjs) runs against report-schema.json and checks that every finding has the required structure. I learned a few things the hard way:

  • The schema expects a flat array, not {"findings": [...]}. First run failed.
  • additionalProperties: false means no extra fields. My id field on each finding? Rejected. Had to remove it.
  • The trace array must start with an entrypoint and end with a sink. The schema enforces this directly.

After the fixes, the validator returned:

Checking [0] HEAD Request Bypasses Dict-Based method_decorators (Authentication Bypass)
Checking [1] abort() and Api.errors Leak Arbitrary Keys into HTTP Response Bodies
Checking [2] Url Field Passes All Object Attributes as url_for() Query Parameters
Checking [3] reqparse Choices List Permanently Mutated When case_sensitive=False

PASS: 4 findings valid

That PASS is satisfying.

Phase 6: Independent verification

This is the part that makes the report credible. Fresh agents with zero prior context read each finding and try to reproduce it with a running PoC.

I wrote a minimal Flask app for each finding:

F-001 PoC — A Resource with method_decorators = {'get': [require_auth]} and a get() method that returns a secret. The test:

GET  /secret → 401 Unauthorized  (auth decorator applied)
HEAD /secret → 200 OK            (auth decorator skipped, bypass confirmed)

The agent confirmed it. Bug is real.

F-002 PoC — A Resource that calls abort(403, secret_key='SK-9f8e7d6c'). The response body:

{"secret_key": "SK-9f8e7d6c", "message": "Forbidden"}

The agent confirmed it. Bug is real.

F-003 PoC — A Resource that marshals an object with password and token attributes through a Url field. The generated URL:

/profile?password=s3cret&token=abc123

The agent confirmed it. Bug is real (though mitigated in practice by ORM models that implement __getitem__).

F-004 PoC — A module-level RequestParser with choices=['Red', 'Green', 'Blue'] and case_sensitive=False. After the first parse, the choices list is permanently lowercase. The id() changes. The agent confirmed it.

All 4 findings survived independent verification with running PoCs. The total time from the Cloudflare blog post to a verified audit was about 90 minutes, with the actual subagent wall time accounting for maybe 30 minutes of that.

The 4 findings

# Finding Severity What it is
1 HEAD bypasses dict method_decorators CRITICAL Authentication bypass on any resource using dict-based decorators
2 abort()/Api.errors leak arbitrary keys HIGH Custom keys passed to abort() appear in JSON response body
3 Url field **data leaks attrs to URLs LOW Object attributes become URL query parameters
4 reqparse choices permanent mutation LOW self.choices permanently lowercased after first parse

F-001 is the one that matters. Any developer who protects a resource with method_decorators = {'get': [auth_required]} finds that GET is protected but HEAD is not. A HEAD request executes get() with full access to request context, including any database writes or side effects, while authentication decorators are skipped entirely. This is a real, exploitable authentication bypass.

The root cause is a logic disconnect in Resource.dispatch_request() at flask_restful/__init__.py:587-603. Method resolution correctly falls back to get() when no head() is defined (correct Flask HEAD semantics). Decorator resolution uses request.method.lower()='head' to look up decorators, which returns an empty list because 'head' is not a key in the dict.

The fix is one line: when HEAD falls back to GET for method execution, the decorator lookup should also fall back to the 'get' key.

How this compares to other approaches

I’ve now run three different multi-agent security audit setups. Here’s the honest comparison:

vs. Anthropic’s defending-code-reference-harness (the canonical one): Anthropic’s harness is designed for C/C++ targets and uses a more elaborate judge-loop architecture with multiple grading passes. It’s more rigorous but slower. For Python targets, the Cloudflare 6-phase approach is leaner and produces findings in a schema that’s easier to consume downstream. The Anthropic version also relies on a find-agent XML-tag submission protocol that feels dated; Cloudflare’s approach treats findings as plain JSON.

vs. a single Claude/GPT agent doing the audit solo: A single agent can do all six phases, but you’ll be paying the same model to argue with itself. The adversarial validation phase in particular is harder to do well with a single agent — there’s a strong bias toward confirming your own prior work. Multi-agent setups with a separate validator role produce a measurably higher false positive rejection rate. In this run, 38% of raw findings were killed by validation. A solo agent would have reported all of them.

vs. running an audit crew of humans: A three-person team with a week on the calendar will find more bugs than twelve agents in an hour. But they will not be schema-validated, and the report will not have machine-checkable traceability from input to sink. The multi-agent approach is not a replacement for human auditors; it’s a way to produce a high-quality first pass that humans can review faster than they could produce it themselves.

vs. a static analyzer like Semgrep or CodeQL: Static analyzers are faster and cheaper but blind to runtime behavior and framework-specific logic. They will not catch the HEAD-vs-GET decorator bypass, because the issue is in how Flask routes HEAD requests combined with a Pythonic dict.get() fallback. That requires understanding the framework’s dispatch semantics, which is exactly what an LLM agent brings.

What I actually learned

The thing I keep coming back to is the validation phase. The Cloudflare pipeline is not special because it uses AI agents. There are a thousand AI agent pipelines for security. What’s special is the discipline of having a phase where the explicit goal is to kill findings.

Most of the AI security work I see is missing this discipline. The agents generate findings, the report goes out, the findings are unverified. Sometimes a finding is “this is a vulnerability” when the actual code path is unreachable. The adversarial validation phase catches these before they become noise in someone’s queue.

The second thing is the schema. Machine-checkable findings with required structure (entrypoint, propagation, sink, conditions, execution, remediation) force a level of rigor that’s hard to fake. The schema can’t tell you whether a finding is real, but it can tell you whether you’ve actually done the work to describe it.

The third thing is the reusability. The whole pipeline is now a Hermes skill called multi-agent-audit-pipeline. It comes with the Cloudflare schema and validator as references. Next time I want to audit a Python library, I run the same six phases against a new target. The agents are the same. The schema is the same. Only the target changes.

The parts I would change

A few things about the pipeline as I ran it:

The schema is strict. additionalProperties: false is a pain to work with. I had to remove an id field from every finding just to pass validation. The strictness is good for machine consumption but annoying for humans adding metadata.

The trace array feels ceremonial. Each finding needs an entrypoint → propagation → sink chain with line numbers. This is useful for audit, but in practice the traces are reconstructed from code reading rather than observed at runtime. A real dynamic tracer that records actual call stacks would be more credible.

The phase 3 validators are still LLM agents. They read code, but they don’t run it. A finding about a specific runtime behavior might be confirmed by code reading or rejected for reasons that don’t actually reproduce. Phase 6 verification with running PoCs fixes this, but Phase 3 itself would benefit from being able to execute test cases.

The wildcard hunter needs better targeting. I got 9 raw findings from the wildcard agent and 7 of them were false positives. That’s a 78% false positive rate for that one role. The agent needs tighter guardrails: “report only if you can write a PoC” would dramatically improve signal.

The meta question

Should you trust an audit done by twelve AI agents in 90 minutes?

Honest answer: the report is as good as what a careful human reviewer would produce from the same evidence. The findings are real. The PoCs work. The schema-validated structure means you can mechanically consume the report.

What’s missing is the contextual judgment a senior auditor brings: is this finding likely to be exploited in the wild? Is the fix what the maintainer would accept? Is this issue in a code path that real users hit? Those questions are still human work.

But the discovery and verification layers are now fast enough that the bottleneck has moved upstream. The question is no longer “can we find the bugs?” but “do we have a maintainer who will fix them when we do?”

That’s a better problem to have.

Reproducing this

If you want to run the same pipeline:

  1. Clone a Python target. (git clone --depth 1 https://github.com/flask-restful/flask-restful)
  2. Read SKILL.md in multi-agent-audit-pipeline. It has the full orchestration recipe.
  3. Spawn 3 recon agents in parallel.
  4. Spawn 3 hunt agents in parallel.
  5. Spawn 3 validate agents in parallel (default to REJECT).
  6. Write REPORT.md.
  7. Write findings.json (flat array, no id field).
  8. Run node validate-findings.cjs findings.json (expects PASS: N findings valid).
  9. Spawn verification agents with running PoCs.

Total wall time for flask-restful: ~90 minutes. Total findings after validation: 4 confirmed, 6 rejected. The skill is now reusable for any Python codebase.

What I read to get here

I wrote this because I wanted to see if the Cloudflare pipeline actually worked outside of Cloudflare’s environment. It does. The parts I’d improve are the schema strictness, the runtime execution of validators, and the false positive rate of the wildcard hunter. The core idea — that the goal of validation is to kill findings, not confirm them — is the part that makes the pipeline work.

Now I have a choice to make. The CRITICAL finding is sitting in findings.json, validated, schema-checked, and independently verified with a running PoC. Whether to report it upstream, write a CVE draft, or sit on it for a while — that’s a decision about disclosure, trust, and timing that no pipeline can make for you.

The pipeline tells you what’s there. What you do with it is still a human call.