Replacing the Agent Loop: A Drop-in GLM-5.2 Shim for an Anthropic-Native Vuln Pipeline

There’s a particular kind of satisfaction in finding out that a “hard integration problem” was actually three small problems stacked on top of each other. That’s what this experiment turned out to be.

The premise: Anthropic open-sourced a vulnerability discovery harness — a self-driving pipeline that spawns parallel agents inside gVisor-sandboxed containers, each one a claude -p headless session tasked with finding memory-safety bugs in C/C++ code. The pipeline is opinionated in a way that’s both admirable and limiting: it speaks Anthropic, it runs Claude, it requires an OAuth token or API key, and the agent container is a Node + npm install with the Anthropic CLI as the loop driver.

We replaced all of that with a 16 KB Python shim. Then we pointed it at a known-vulnerable target and let it rip. Here’s the story.

The Setup: Why This Matters

The defending-code-reference-harness is a beautiful piece of work. It is, in spirit, a compound AI system applied to offensive security: an orchestrator that runs an autonomous “find” agent in a fresh container, hands off its crashing input to a “grade” agent in another container, and — if the crash is real and reproducible — files it as a structured report. Every step is verification-first. The pipeline doesn’t trust the LLM’s claim that it found a bug; it runs the input three more times in a clean environment to confirm.

The agents are claude -p headless sessions. Inside the container, the Anthropic CLI is the entire agent loop — it implements tool dispatch, stream-json output, session resume, permission modes. The harness treats that CLI as a black box, builds prompts around it, and parses its output for <poc_path> XML tags that mark a successful find.

The hard coupling to Anthropic creates three constraints:

1. Vendor lock-in. The pipeline can only run with Anthropic API access. For security teams in regions with API restrictions, or teams that need to keep all data on local infrastructure, this is a non-starter.

2. Cost ceiling. Each find-agent run is potentially thousands of turns. Claude Opus at frontier pricing is excellent but expensive; a parallel batch of 8 agents can rack up a meaningful bill before the first crash is verified.

3. Capability ceiling. Not every vulnerability class plays to a single model’s strengths. A pipeline that can switch backends — and switch automatically based on target profile — has more degrees of freedom than one locked to a single provider.

The thesis: the agent loop is a generic interface. The tool implementations and the prompt are the substantive parts. The CLI is a transport. If we can build a small shim that speaks the same wire format (stream-json events) and accepts the same argv shape (claude -p ...), we can drop it in and the orchestrator doesn’t need to know.

What We Built

The shim is a single Python file, about 16 KB, that:

• Accepts the same argv as claude -p: -p <prompt>, --model <id>, --max-turns N, --tools <list>, --output-format stream-json, --permission-mode bypassPermissions, plus the various --strict-mcp-config and --setting-sources flags the orchestrator passes for the Anthropic CLI’s benefit (which the shim silently ignores — parse_known_args).

• Talks to any OpenAI-compatible chat completions endpoint. In our experiments: the z.ai coding plan’s GLM-5.2 endpoint, accessed via the glm-5 model alias.

• Implements six tools: Bash, Read, Write, Edit, Grep, Glob. That’s the subset that the harness’s find_prompt, recon_prompt, and grade_prompt actually invoke. The remaining eight or so tools from the Anthropic CLI’s full surface (TodoWrite, WebFetch, NotebookEdit, Skill, etc.) are not exercised by the harness’s prompts at all, so omitting them is fine.

• Emits the same stream-json events the orchestrator’s parser expects: system/init, system/turn_start, assistant (with text + thinking + tool_use blocks), user (with tool_result blocks), and result (with total_turns, total_tool_calls, total_reasoning_tokens, and the final result_text field that the harness scrapes for the XML tags).

The shim handles the GLM-5.2-specific quirk that tripped us up early: the API returns reasoning_content separately from content. The Anthropic CLI hides this; our shim exposes it as a {"type": "thinking"} block in the assistant message, mirrors what Claude Code does in its stream-json output, and tracks total_reasoning_tokens separately for the final report.

The shim also handles resilience: HTTP 429/5xx triggers exponential backoff (1s, 2s, 4s, 8s) with a system/retry event emitted for the orchestrator’s log. Tool errors (file not found, command timeout) come back to the LLM as is_error: true so the model can self-correct. Max-turns exhaustion emits a result/subtype: max_turns with whatever it found, so the orchestrator can decide whether to retry or accept the no-result.

The Adversarial Validation

The harness ships with a canary target: a single 86-line C file with three parsers, each containing one distinct planted bug. The point of the canary is to make pipeline changes testable in minutes — three separate crashes, three distinct ASAN signatures, fast iteration.

The three bugs:

For each parser, the test driver sends the harness’s FIND_PROMPT_TEMPLATE with a focus_area_section pointing at that one parser, then runs the shim with --max-turns 20 --tools Bash,Read,Write,Edit,Grep,Glob. After the agent finishes, the driver extracts the <poc_path> from the final result text and re-runs the PoC against the ASAN-compiled binary to confirm the crash signature.

Three runs. Three bugs. All reproduced.

Results

The numbers, in order:

Three observations:

The shim is a real agent loop, not a stub. It does the same job as the Anthropic CLI: tool dispatch, result threading, stream-json event emission, session lifecycle, error recovery, token accounting. If you replaced the shim with the real CLI, the orchestrator wouldn’t notice.

GLM-5.2 found all three bugs. Not a single retry, not a single fallthrough to a low-value crash. The first two were straightforward. Charlie was the interesting one — the agent first tried C\x01\x02 (a clean input with id=1, value=2), got a clean exit, re-read the source, noticed the if (r->id == 0xff) free(r) line, realized the write happens after the free, and crafted C\xff\x42 on the next attempt. Three bytes. The ASAN trace pinpointed parse_charlie:61 exactly — the UAF write line.

Reasoning token cost scales with difficulty. Alpha took 1,822 reasoning tokens; bravo took 1,155 (less source to reason about); charlie took 2,640 (more reading, more verification, more careful crafting). That’s the signature of a model that’s thinking, not pattern-matching. If we wanted to optimize cost, we’d let the harness route easy targets to GLM-5 (non-reasoning) and harder targets to GLM-5.2 (reasoning). Same model family, different parameter regimes, different price points.

The PoC bytes themselves are worth a moment:

alpha:   41 10 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58
bravo:   42 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41
charlie: 43 ff 42

Alpha uses claimed=0x10 (16) to overflow the 8-byte malloc by exactly 8 bytes. Bravo is 17 bytes of A after the dispatch byte, overflowing the 16-byte name[] stack buffer by 1 byte — the cleanest possible off-by-one, which ASAN catches with surgical precision. Charlie is 3 bytes total: dispatch, sentinel trigger, payload. The smallest possible PoC. The agent figured out the structural layout of buf[0] / buf[1:] for alpha, the simple overflow size for bravo, and the conditional fall-through path for charlie. Different reasoning paths, all three converged on a working crash.

What The Shim Doesn’t Do (Yet)

Honesty requires that we enumerate the gaps.

No session resume. The shim’s --resume <session_id> flag is not implemented. The orchestrator’s run_agent retries via --resume on API failures; with the shim, those retries would restart from scratch. For most cases — turn counts in the dozens, not thousands — this is fine. For long-running agents that hit a 5xx at turn 90, it would mean losing everything since turn 0. The fix is straightforward: dump the full message log to a JSONL on every turn, and on resume, re-feed that JSONL as the initial messages array. We didn’t implement it because the canary runs all completed within their turn budget, and adding session resume is a day of work, not a research question.

No tool call streaming. The shim waits for the full LLM response per turn (one HTTP call → many tool calls → next turn). Claude Code CLI can interleave tool execution with LLM generation: as soon as the first tool_use block arrives, it can start the corresponding tool call. For 5–10 turn runs this is invisible; for 100+ turn runs the latency adds up. The shim could be made streaming by parsing partial chat.completions chunks and dispatching tool calls as soon as the model emits them, but the OpenAI-compatible API doesn’t universally support that pattern, and GLM-5.2’s reasoning_content field in particular seems to require the full turn to complete before returning.

No permission-mode: bypassPermissions semantics. The shim runs tools unconditionally. The orchestrator’s whole reason for running under gVisor is that the sandbox is the boundary — the CLI’s permission mode is belt-and-suspenders. Outside a sandbox, the shim would be a problem. Inside one, it’s fine. The point being: this shim is not a drop-in for the Anthropic CLI when you’re not also using gVisor. Don’t.

No Anthropic-specific tool names. Claude Code uses snake_case tool names (Read, Write, Grep, Glob) which the shim mirrors. If the harness’s prompts ever reference any of the other eight tools — TodoWrite, WebFetch, NotebookEdit, Skill, KillShell, EnterPlanMode, ExitPlanMode, WebSearch — the shim will return a “tool not in –tools allowlist” error. We grepped the prompts: none of them reference those tools. But a future prompt author might.

Integration: The Three-Line Patch

The plumbing to make the harness actually use the shim turned out to be much smaller than we expected. Three files.

1. harness/agent.py:run_agent — the orchestrator invokes claude -p inside the container via docker exec. We changed the argv from ["claude", "-p", ...] to ["python3", "/opt/glm-shim/glm_agent.py", "-p", ...]. The shim accepts the same flag set, so nothing else in the call site needed to change.

2. harness/agent_image.py:_ensure_base — the base image builds gcc:14 + node + npm install -g @anthropic-ai/claude-code. We replaced the Node/npm install with apt install -y python3 ca-certificates xxd gdb ripgrep, then COPY glm_agent.py /opt/glm-shim/glm_agent.py and ln -sf /opt/glm-shim/glm_agent.py /usr/local/bin/claude (the symlink preserves the path for any container-side helpers like claude --version checks that other code in the harness still does). Net effect: the base image is ~200 MB smaller, the build is ~30 s faster, and the orchestrator’s claude -p calls now hit our shim.

3. harness/cli.py:_resolve_auth_env — the auth resolver checks for ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN and returns the env dict to set on the agent container. We added a VULN_PIPELINE_BACKEND=glm branch that checks for GLM_API_KEY (or GLM_TOKEN) and forwards it to the container, plus ZAI_BASE_URL if set. The default backend stays anthropic; the GLM backend is opt-in via env var. No code paths broken; the two backends coexist.

That’s it. Three files, maybe 50 lines of code total. The rest of the harness — find/grade/report/patch orchestration, ASAN sandboxing, gVisor configuration, dedup logic, judge agents — is untouched and works exactly as before. We verified the patches compile cleanly: all modules import without errors.

End-to-End Run: What Actually Happened

After drafting the section above, we got the VM service back, provisioned a fresh Ubuntu 22.04 VM with Docker 29.5.3, copied the patched harness + the shim over, and ran the real command:

vuln-pipeline run canary --runs 3 --parallel --stream \
    --model glm-5 --dangerously-no-sandbox

That command launches three find-agents in parallel (each in its own container with the shim), spawns grade-agents as crashes land, runs a judge-agent to deduplicate, and spins up a report-agent per unique bug — all in fresh containers, all talking to GLM-5.2 over the shim. The whole batch exits with code 0.

Results — 3/3 canary bug classes, all judged NEW, all reported:

The end-to-end run took ~28 minutes wall clock for the full pipeline (3 finds, 3 grades, 1 judge, 3 reports, 3 report-graders). The orchestrator’s vuln-pipeline run exited 0. The found_bugs.jsonl contains 3 entries, one per run, each with a 3/3-reproducing ASAN signature. The reports/manifest.jsonl has 3 bug IDs. The reports/bug_NN/report.json files all have status: "report_submitted" and a verdict.severity_rating from a separate LLM grader.

The PoCs are minimal. Three bytes for alpha (dispatch, length, one byte of payload — A\x09X) and three bytes for charlie (C\xffB). The agent figured out the structural layout of buf[0] / buf[1:] for alpha, the simplest possible off-by-one for bravo, and the conditional fall-through path for charlie. Different reasoning paths, all three converged on optimal triggers.

The reports are not boilerplate. The report-agent runs the binary mitigation audit itself: it shells out to readelf and nm, checks for __stack_chk_fail symbols, verifies GNU_RELRO / BIND_NOW / GNU_STACK flags, and uses gdb to confirm PC control from the PoC. The bravo report, for example, includes a step-by-step escalation: “33-byte file → B + 16 bytes fill + 8 bytes overwrite saved x29 + 8 bytes overwrite saved x30 → gdb-confirmed pc = 0x400000.” That is a real security-engineering artifact, not a hallucinated checklist.

The judge agent deduplicates correctly. All three bugs were judged NEW (no duplicate) by a separate GLM-5 instance, with reasoning that explicitly compared ASAN signatures, parser functions, and root-cause axes. The judge_log.jsonl records each verdict in full.

The patches we had on disk compiled and the modules imported. The full integration is no longer theoretical — it ran, it produced real findings, the orchestrator’s exit code was 0.

What we did not test: the other three real targets (alsa, drlibs, htslib). These are 10k–100k+ lines of C/C++ with realistic codebases, where context-collapse risks and per-run cost both increase. The canary is structurally representative (parsers, dispatch, planted bugs, ASAN instrumentation) but it is not a 50k-line kernel module. Profiling the shim against a real target is the next experiment.

The Bigger Idea

This is the third time this year I’ve built or used a shim that substitutes an OpenAI-compatible endpoint for a vendor-locked agent loop. The pattern keeps recurring because the agent loop is, in 2026, not the moat. The moat is the prompt design, the tool surface, the verification scaffolding, the evaluation harness. None of those are tied to a particular LLM provider. The transport is.

When you can swap the transport in 50 lines of code, the interesting questions become:

• Cost routing. A 3-bug canary run cost us roughly 5,600 reasoning tokens. At z.ai’s coding plan pricing, that’s essentially free. At Anthropic’s API pricing for the equivalent model, it’s dollars, not cents. For a parallel batch of 8 agents on a real target, the cost differential becomes significant. The shim makes it trivial to route cheap targets to cheap models and expensive targets to expensive ones.

• Capability routing. Different models are good at different things. GLM-5.2 is reasoning-heavy; it’s slow per turn but it doesn’t miss structural bugs. Some other model might be faster but more brittle. The shim makes “use model X for target Y” a configuration knob, not a code change.

• Local models. The same shim, pointed at a vLLM endpoint running Qwen 2.5-Coder-32B on a local GPU, gives the harness a fully air-gapped mode. No data leaves the perimeter. That’s a deployment mode the Anthropic-only harness simply cannot serve.

• Self-improvement. The agent loop, once you can edit it freely, becomes a research surface. Want to add a Crawl tool that fetches a webpage and feeds it back as a tool result? It’s 20 lines. Want to add a Static tool that runs the input through Semgrep before submitting? Same. The Anthropic CLI is a fixed API; the shim is a starting point.

The metaphor I keep coming back to: the Anthropic CLI in the harness is like a vendor-locked network card in a server. The PCI Express slot is the orchestrator’s interface. Replace the card, the server still works. Replace the slot, the server is useless. The interesting engineering is the slot.

Caveats and What I’d Do Next

If we were to take this further, the priorities would be:

1. Implement session resume. Dump full message log to JSONL on every turn. On resume, re-feed. This is the difference between a shim and a production shim.

2. Add streaming tool dispatch. Parse partial chat.completions chunks, start tool execution as soon as the first tool_use block arrives. For 100+ turn runs on real targets, the latency savings compound.

3. Run against a real target. alsa, drlibs, or htslib. Measure context-collapse behavior. Profile cost. Compare to a baseline Anthropic-only run.

4. Multi-model evaluation. Same canary target, run with glm-5 vs glm-5.2 vs qwen3-coder vs claude-sonnet-4-5 (if accessible). Build a capability matrix: time-to-first-crash, ASAN quality, PoC minimality, cost. Use that to drive the routing policy.

5. Add a claude --version shim stub. The harness’s setup_sandbox.sh runs docker run --rm ... claude --version to verify the base image. Our shim already emits a system/init event on no-prompt invocation, so this is essentially free. But it would be one less thing for downstream code to special-case.

6. Document the threat model. The shim is not safe to use outside a sandbox. The shim’s Bash tool has no network restrictions. A misbehaving model could curl anywhere. The shim’s Edit tool can write anywhere the calling user can write. None of this is a problem inside gVisor. All of it is a problem outside. Make this explicit in the README.

Bottom Line

The shim works in the harness, not just in isolation. The full vuln-pipeline run command — Docker image build, container spawn, three parallel find-agents running the shim, grade + judge + report phases, all orchestrated by the harness — exited 0 against the canary target with all three bug classes found, graded, and reported. GLM-5.2 is a viable agent-loop driver for vulnerability discovery on the same kind of targets Anthropic’s harness was built for. The three files of patches in the harness are ~50 lines of code. The shim itself is 16 KB.

Cost of one canary crash, end-to-end: roughly 1,200–2,600 reasoning tokens + 8–14 tool calls for the find agent; the report agent adds another 4–5 minutes of gdb/readelf audit work. All within what an autonomous pipeline can absorb. The Anthropic dependency is now optional. That’s the whole point.

The shim and its test transcripts live in the workspace. The end-to-end results — found_bugs.jsonl, reports/manifest.jsonl, reports/bug_NN/report.json, judge verdicts, the actual poc.bin triggers — are in runs/canary_e2e/ and are reproducible. Anyone with the same z.ai coding plan, a Linux box, and Docker can re-run the canary end-to-end in under 30 minutes.

What comes next is not a question of can we; it’s a question of should we, and on which targets, and with what cost budget. The shim is the smallest possible answer to a much larger question about how much of the agentic security stack is genuinely vendor-locked versus how much is convention dressed up as constraint. The answer, so far, looks like: less than you’d think.