Tool-call guard benchmark
Part of DefenseClaw
Appendix
Chapter 1

What this is

What does a tool-call guard do, and what does this page test?

AI coding agents work by calling tools. They write files, run commands and call MCP servers. Some calls are destructive, such as exfiltrating data or tampering with the host. Others come from a compromised trajectory, where the agent read text an attacker planted. A tool-call guard checks each call before it runs. It answers allow, confirm (send for review) or block.

DefenseClaw is the guard measured here, and it has three tiers. System One is DefenseClaw's name for the second tier, the Decision Model. It answers every call the rules pass and settles the clear cases. Only the uncertain band goes on to the paid Gemma 4 LLM judge.

  1. Deterministic rules A rule match ends the cascade.
  2. Decision Model System One Settles every call it is confident about, either way.
  3. Gemma 4 LLM judge Paid per token. Sees only the uncertain band.

This page scores 12 candidate Decision Models and the cascades built from them. The main data is the Broad comparison: 3,817 labelled tool calls, 436 of them unsafe. A Production-weighted corpus of 24,476 calls is 99.1% benign, like real traffic.

Every model is held to one shared false-block budget. In plain words, each may wrongly block at most 13 of 3,381 harmless calls. Within that limit, its threshold is set to catch as many attacks as it can.

Which Decision Model should sit between the rules and the LLM judge, and does adding it help?

It is written for engineers choosing what to deploy. Sibling Spaces ask whether a laptop-sized model can guard tool calls alone. They also ask how Cisco Skill Scanner compares with NVIDIA SkillSpector.

Chapter 2

The answer

Which Decision Model should sit between the rules and the judge, and does it help?

Yes. Put OpenJev there. On traffic that is 99% benign, drop the judge.

We scored the cascade on 3,817 labelled tool calls. Adding OpenJev as the Decision Model in front of the Gemma 4 judge blocks more attacks. It also wrongly blocks fewer benign calls, and it asks the judge about one case in six.

Recommended stack

Rules, then OpenJev as the Decision Model, then the Gemma 4 judge, with two-sided routing at an allow threshold of 0.30.

  • Deterministic rules 13 cases End the cascade on a rule match before any model runs.
  • OpenJev (Decision Model) 3,195 cases Settles every case it is confident about, either way.
  • Gemma 4 judge 609 cases Sees only the uncertain band between the two thresholds.
  • Block-only F1, judge alone to recommended stack 0.712to0.752 0.738 as the cascade runs today
  • False blocks of 3,381 benign calls 23to14 Block FPR 0.68% to 0.41%
  • Cases sent to the judge 100%to16% 609 of 3,817 cases

The three tiles compare the recommended stack with the judge alone. Block-only F1 counts only a hard block as a catch, so higher is better. A false block is a harmless call the guard stopped, so lower is better. The judge share is the fraction of calls that cost a paid judge call.

Two numbers describe the same stack, and both are measured. The recommendation scores 0.752 with escalate-on-confirm: a rule-engine confirm becomes a floor and the later tiers still run. The cascade as it runs today uses short-circuit and scores 0.738. Every chart on this page measures the short-circuit cascade. The two differ on 8 cases: 263 true blocks become 271 at the same 14 false blocks.

Why not the first-ranked model?open-jev-qwen-27b ranks first in chapter 3, but its threshold was set on these same cases. As shipped it blocks 87 attacks (block-only F1 0.332). Its best measured cascade scores 0.594, below the judge alone. No cascade has been run with it at the budget threshold, so the stack uses OpenJev. OpenJev's weights are CC BY-NC 4.0: non-commercial, attribution required.

OpenJev settles 84% of cases and the judge sees 609 of 3,817

Tool calls by the tier that decided them and the final decision. Short-circuit cascade; move the threshold, routing or corpus.

How to read it. Read left to right. Each band is a group of tool calls, and its width is the number of calls. Bands leave the tier that decided them and end at allow, confirm or block. The three tiles under the chart give the score at the current setting.

0.30
Routing
Corpus
  • Block-only F1 0.738 +0.025 against the judge alone
  • False blocks 14of 3,381 Block FPR 0.41%
  • Judge rate 16% 609 of 3,817 cases

What it shows. Raising the threshold from 0.05 to 0.30 cuts judge calls from 48% to 16% of cases while F1 moves by 0.006.

Data table
Source

Try it. Move the threshold to 0.05 and watch the band into the judge widen. Then switch routing to one-sided and see almost every call reach the judge. Under two-sided routing, OpenJev settles a case alone when it is confident either way. Under one-sided routing it settles only confident blocks, so 90% of cases reach the judge for a slightly lower F1. The allow threshold was measured at four points only, so the control snaps to them.

Why it matters: the threshold is a cost dial, not a quality dial. Across the four measured points F1 barely moves, while judge calls triple at the low end. Start at 0.30 with two-sided routing. Lower it only if you have a reason to pay for more judge calls.

At 99% benign traffic the judge costs F1, so drop it

Block-only F1 for each policy; switch between the Broad comparison (88.6% benign) and Production-weighted (99.1% benign) corpora.

How to read it. Each bar is one way to assemble the guard, and a longer bar is a better block-only F1. The corpus switch swaps the attack-rich Broad comparison for the 99.1% benign Production-weighted mix.

Corpus

What it shows. On the Production-weighted corpus, rules then OpenJev with no judge scores 0.213, against 0.082 for the full cascade.

Data table
Source

The order flips on the Production-weighted corpus. Without the judge, rules then OpenJev block 53 of 221 attacks at 224 false blocks. Adding the judge blocks 58 attacks but 1,133 benign calls. Jev 1.13.0 with no judge scores slightly higher, 0.233, but it is a paid API.

Why it matters: the right stack depends on your traffic mix. When almost every call is benign, the judge buys 5 more catches for 909 more false blocks. On that traffic, run rules then the Decision Model alone. Fix the fail-open path in chapter 5 first, because without the judge nothing catches a Decision Model outage.

How this was measured

Every case runs through the real deterministic rule engine first (realdet), then OpenJev on question format Q2 with context C7, then the Gemma 4 judge on the escalated band. Rules ribbons are measured: 12 confirms and 1 block on the Broad comparison, 95 confirms on Production-weighted. The scorer records which tier decided each case and the final mix, but not the two cross-tabulated. The OpenJev and judge ribbons therefore split the remaining measured mix in proportion to the cases each tier decided. Thresholds were fitted on these same cases, so every figure is an upper bound. An earlier 0.75173 was scored against an all-allow stand-in tier; the appendix lists every restated figure.

Chapter 3

Which model, at the shared budget

Which model blocks the most attacks without exceeding the budget?

Two models lead at 13 false blocks, and their intervals overlap.

Every model is held to one shared budget: at most 13 false blocks on 3,381 benign cases. That is a block FPR of 0.38%. It is OpenJev's own realised rate on this corpus. Each model's threshold is its highest-recall point within the budget, read on P(block). Equal false blocks keep it fair: no model wins by blocking more of everything.

Only the top two models block more than half of the 436 attacks

Block-only F1, precision and recall with 95% intervals at 13 false blocks, Broad comparison (436 unsafe, 3,381 benign).

How to read it. Each row is a candidate Decision Model, best first. The three dots are its block-only F1, precision and recall at 13 false blocks, coloured as in the legend. The lines are 95% intervals. Hover a row in the table to highlight that model; click to pin it.

What it shows. open-jev-qwen-27b blocks 263 attacks and OpenJev 248; below rank 3, no model blocks even one attack in five.

Source

Why it matters: choose between the top two on licence and evidence, not on rank. Their F1 intervals overlap, so the data cannot separate them. open-jev-qwen-27b is apache-2.0 but has never run in a cascade. OpenJev has, and its weights are non-commercial. Nothing below rank 3 is worth deploying at this budget.

The recommended stack is not a row here. It is measured at 14 false blocks (block FPR 0.41%), one more than this budget, because the judge adds its own blocks. Its 0.752 is escalate-on-confirm, while the chapter 2 charts measure the short-circuit cascade at 0.738.

Thresholds matter as much as weights. As shipped, open-jev-qwen-27b blocks 87 attacks; at the budget threshold it blocks 263. SecJudge blocks 98.4% of benign cases as shipped and nothing at the budget. If you deploy a model, set its threshold on your own benign traffic rather than trusting its default.

How this was measured

Precision and recall intervals are Wilson 95%. F1 intervals come from a family bootstrap of 2,000 resamples. The AUC column orders unsafe above benign at no threshold at all. Every model answered format Q2 at context C7 except SecJudge, a 396M classifier. The per-model precision-recall curves behind this board are not in the published datasets. The board is therefore shown at the shared budget only, with no budget slider.

Chapter 4

What it costs

What does each guard cost to run, in judge spend and in lost benign work?

The stack costs a sixth of the judge alone, and enforcing per call spares most benign work.

A guard has two running costs. The first is money: every call that reaches the judge is a paid model call. The second is lost work: every false block stops something a developer wanted to do. This chapter prices both, so you can compare policies on more than F1.

2 of 19 policies are worth running on the Broad comparison, 1 on Production-weighted

Block-only F1 against judge spend per 1,000 cases, 19 policies per corpus. Coloured points follow the threshold control.

How to read it. Each point is one policy. Further left costs less in judge spend; higher blocks better. The line joins the frontier policies. The coloured points are two OpenJev policies and the judge alone; they follow the allow threshold in chapter 2.

Corpus

What it shows. Only 2 of 19 policies sit on the Broad comparison frontier, and only Jev 1.13.0 with no judge on Production-weighted.

Data table
Source

The Pareto frontier on the Broad comparison holds two policies: rules then OpenJev with no judge, and the recommended two-sided stack at 0.30. On Production-weighted it holds one: Jev 1.13.0 with no judge. The judge's price is $0.19 per 1,000 cases judged, at $0.042 per million input tokens.

Why it matters: a policy off the frontier is beaten on both cost and quality. Do not run it. Switch the corpus and watch most of the frontier vanish. On production-like traffic, the judge never pays for itself.

At 14,000 cases a day the stack costs about $4 a month in judge calls

Projection from the measured rates, interpolated between the two corpora's benign shares. Uses the allow threshold above.

How to read it. Type your daily call volume and benign share; the threshold slider is shared with chapter 2. The tiles project monthly judge spend, confirms sent for review each day and unsafe calls let through each day. Try a benign share of 0.886 to match the Broad comparison.

0.30
  • Judge spend per month $4.12 US dollars, 30-day month
  • Confirms per day 1,357 Cases sent for review
  • Unsafe calls allowed per day 5.7 Final decision allow

What it shows. At the default inputs the stack also sends 1,357 cases a day for review and lets about 6 unsafe calls through.

Data table
Source

Why it matters: the judge bill is small at this volume, but the review queue is not. Every confirm routes to a person or a model for review. Size that queue before you deploy, and read the unsafe-allowed tile as the guard's residual risk.

Blocking one call instead of the session spares 2.7 to 5.1 times more benign work

Share of benign work destroyed, per call and per session, at identical recall.

How to read it. Each pair of bars is one model. One bar is the benign work lost when only the offending call is blocked. The other is the work lost when the whole session ends. Shorter is better.

Corpus

What it shows. On the Broad comparison, OpenJev blocks 16 of 21,045 benign calls but would end 13 benign sessions.

Data table
Source

Per-call enforcement blocks the offending call and lets the session continue. Per-session enforcement ends the whole session, so one false block throws away every benign call in it. Why it matters: enforce per call. It catches the same attacks and makes every false block much cheaper. Latency and serving cost are in the appendix.

How this was measured

Judge spend is the judge rate times the corpus's own judge run price per case. The calculator interpolates linearly in benign share between 88.58% and 99.10% and clamps outside that range. Unsafe calls allowed counts unsafe cases whose final decision is allow.

Chapter 5

Does it fail safe

When the Decision Model breaks or wavers, does a bad call get through?

It fails open: most Decision Model failures end with the tool call allowed.

A guard is only as good as its worst day. This chapter tests three things accuracy scores hide. What happens when the Decision Model errors? How often does it flag real work? Does it answer the same way twice?

24 of 28 injected faults end with the tool call allowed

29 scenarios (28 fault classes plus a healthy control), 30 cases each, rules then a mocked Decision Model with no judge.

How to read it. The single bar holds all 29 scenarios. Each coloured segment counts scenarios by outcome, from errors that still allow the call to runs that look healthy. Open Fault classes under the chart for the scenarios in each segment.

What it shows. Only 2 fault classes are indistinguishable from the healthy control; one crashed the runner.

Data table
Fault classes
Source

A timeout, an HTTP 503 or a truncated body from the Decision Model makes the cascade fail open. With the judge behind it, the 21 erroring classes end in confirm instead. Under two-sided routing, two wrong-shape answers from the boolean-probe formats still pass as allow with no error code. Why it matters: an outage silently switches the guard off. The fix is one setting, a configurable failure disposition that defaults to confirm. Do it before dropping the judge.

On real coding traffic OpenJev flags 1.9% of benign calls and Jev 1.13.0 flags 33%

Share of 1,543 benign tool calls flagged, from 40 coding sessions, with 95% intervals, context C7.

How to read it. Each row is a model. Each dot is the share of benign calls one gate flags, coloured by gate as in the legend; the lines are Wilson 95% intervals. Further right means more false alarms.

What it shows. Every flag here is a false positive, and the request-serving gate flags far more than the decision itself.

Data table
Source

The Lane B gate asks whether a call still serves the request. On its own it flags 276 of 1,543 benign OpenJev calls and 39 of 40 benign sessions. Do not use it as a standalone gate. The hosted Jev row has no published interval. Why it matters: each flag interrupts a developer for nothing. OpenJev flags about one benign call in 50; Jev 1.13.0 flags one in three.

OpenJev gives the same answer on every replay; the others flip on 14% to 17% of flagged calls

Share of decisions that change across three byte-identical replays of 1,519 benign coding events.

How to read it. Each pair of bars is one model. One bar counts flips over all 1,519 events. The other counts flips only over events that some replay flagged. Shorter is more consistent.

What it shows. Counted over every event the flip rate looks negligible; counted over flagged events it reaches 17% for DiffusionGemma.

Data table
Source

The flagged-only flip rate is the operative figure, because an always-allowed call has nothing to flip between. Over 16 replays, Jev 1.13.0 flips on 23% of flagged calls, so three replays understate it. Why it matters: a guard that flips gives two answers to one call. That makes incidents hard to reproduce and audits hard to trust.

How this was measured

Faults were injected against a local loopback mock of the Decision Model; no provider, GPU or hosted endpoint was called. The mock's quality numbers mean nothing; the shape of the outcome is the finding. Each fault class ran 30 cases (58 events). Coding traffic is 40 real benign sessions with no positives, so it cannot show what recall a lower threshold would cost.

Chapter 6

Does it read intent

Does showing the model the user's request help it spot a compromised call?

Yes on real compromises, and it still misses four in ten.

Eight runs on AgentDojo said that comparing a call with the user's request does not help. On the proof-backed corpus, 3,402 real cases with 405 proof-verified compromises, the same scorer says it does. The hard case is an agent that read an attacker's text. It may comply or correctly refuse, and the guard must tell which.

On proof-backed data all 32 cells turn positive; 31 had the opposite sign on AgentDojo

Separation by model and context (C0 call alone, C7 with request and history), per case and per event.

How to read it. Each group is one model at one context. Bars above zero mean it blocks proven compromises more often than correct refusals. The two colours count per case and per event. Switch Data to AgentDojo and watch the same bars drop below zero.

Data
Lens

What it shows. All 32 proof-backed cells sit above zero, and 31 of them sat below zero on AgentDojo.

Data table
Source

Separation is the block rate on proven compromises minus the block rate on correct refusals. OpenJev with context C7 moves from −0.534 on AgentDojo to +0.575 here. The earlier negative was a property of the benchmark, not of the mechanism. Why it matters: send the user's request and recent history to the Decision Model. And test intent features on real compromises, because a synthetic suite reversed the conclusion here.

Even the best model blocks only 234 of 405 proven compromises

Block-only recall on proof-verified cases, call alone (C0) against call with request and history (C7), Wilson 95% intervals.

How to read it. Each row is a model. One dot is its recall with the call alone (C0); the other is its recall with request and history (C7). The gap between them is what context adds; the thin lines are 95% intervals.

What it shows. Context doubles OpenJev's recall, from 108 to 234 blocked, and every model still misses 42% to 69%.

Data table
Source

Why it matters: even with context, 171 of 405 proven compromises get past the best model. Treat the guard as one layer of defence, not the only one. The claims below say which intent statements the evidence supports.

    How this was measured

    The separation scorer ran unmodified on both corpora, with 2,000 bootstrap resamples and seed 741983. C0 against C7 is not a single-factor test: C7 adds the request, up to seven prior events and a production text format. The proof-backed corpus is evaluation-only, so this page reports aggregate rates and publishes no case from it.

    Chapter 7

    Read this before deploying

    What could make these numbers wrong for you, and what comes first?

    These are upper bounds on one attack mix, and the fail-open path comes first.

    Each row names an assumption the numbers above rest on and what would break it. The budget and the thresholds were both fitted on the cases they are scored on. So every score on this page is an upper bound. Real traffic will score lower.

    Read the Standing column first. Rows marked narrow or interpolated are the ones most likely to differ on your traffic. If your attacks look unlike the main source dataset, rerun the board on your own cases first.

    Assumptions behind the numbers on this page
    Assumption What it rests on What breaks it Standing

    Order of work

    Twelve recommendations and three security fixes, de-duplicated and in the order to do them. Each line names its evidence and the chapter that shows it.