Risks

What goes wrong around the model: failures that resolve to allow, benign work flagged as unsafe, answers that change on replay, and the cases where the deciders disagree.

Failure handling

Twenty-eight failure classes were injected against a local mock: refused and reset connections, timeouts, HTTP 429, 500, 502, 503 and 422, truncated bodies, invalid or duplicate JSON, missing, wrong or extra answer keys, out-of-range probabilities, wrong value types and a deliberate silent allow.

Injected fault classes and their outcomes

Every network, HTTP, truncation, JSON and schema failure that can plausibly happen was injected against a local mock. The quality numbers from that mock are meaningless; the shape of the outcome is the finding.

Model errors, cascade allows anywayModel answers in the wrong shape, cascade allows, no error codePartially degradedIndistinguishable from healthy (includes the control)Crashed the runner, no prediction file written

Data table

Outcome group	count	scenarios
Model errors, cascade allows anyway	21	connection_refused, connection_reset, connection_reset_mid_response, duplicate_json_keys, extra_answer_key, hard_timeout, http422, http429, http429_retry_after, http429_retry_after_date, http500, http502, http503, invalid_json, missing_answer_keys, object_choice, out_of_range_probabilities, truncated_body, truncated_json, type_mismatch_q0_choice, wrong_answer_keys
Model answers in the wrong shape, cascade allows, no error code	3	silent_allow, type_mismatch_q1_noul, type_mismatch_q3_noul
Partially degraded	1	intermittent_20pct_http503
Indistinguishable from healthy (includes the control)	3	extra_top_level_key, intermittent_20pct_truncated_json, ok
Crashed the runner, no prediction file written	1	nonfinite_usage

21 classes write an error row and the cascade allows anyway; 3 answer in the wrong shape and are allowed with no error code; 3 are partially degraded; 1 crashes the runner on a non-finite token count and writes no prediction file. With the judge behind the small model every all-allow class becomes confirm instead, because all 11 of the judge’s own injected failure modes answer confirm. Without the judge, which is the recommendation at production traffic, the allow is what ships.

Q1 and Q3 fail open

Q0, Q2 and Q4 answer with a disposition, and a value outside the vocabulary returns action = "error". Q1 and Q3 derive an action from eight boolean slots with probabilities.get(name, 0), so a response that parses but carries no signal becomes allow at confidence 1.0 with no error code. That is indistinguishable from a confident correct allow, and no scorer downstream can see it.

The guard that closes it:

if not any(key.endswith(".true") for key in probabilities):
    return "error", 0

It never fired in the measured runs: 0 of 90,930 scored rows across 3 audited runs. The defect is latent, and Q1 and Q3 also block too rarely to be used for enforcement (Method). A second, smaller defect: when two dispositions tie to within 1e-9, the winner was whichever key the provider emitted first; the fix resolves the tie toward the higher severity.

The promotion gate

A candidate can fail every diagnostic request and still be promoted. The fix is to read the prediction file’s own error rows. The gate does work on scorable rows: at the Production-weighted stage it rejected OpenJev for 32 provider errors in 100,001 requests.

Resume validation

Long runs resume from a partial prediction file. Of 11 tampers probed, 5 were accepted (counting the untampered control): flip_action_to_allow, flip_action_to_block, inject_error_code, corrupt_context_sha256. The run then reports complete. The validator checks which request each row answers, not what the answer was. A resumed run also keeps its predecessor’s errored rows unless --resume-retry-errors is passed.

Benign traffic

The serves-the-request score as a gate

The Q4 format returns a second score, serves_intent (“does this still serve the user’s request?”), called Lane B here. As a standalone gate at its nominal 0.5 threshold it flags 0.17887 of benign events for OpenJev and 0.10110 for DiffusionGemma, 3.7× to 9.2× each model’s own full-decision rate, and 39/40 and 31/40 of the 40 sessions. On OpenJev the score never exceeds 0.8223 on any benign event, so no threshold separates the traffic. This corpus has no unsafe cases, so the recall cost of a lower threshold is unmeasured.

Flag rates on benign coding traffic

40 real coding-agent trajectories, 1,543 tool-call events, 100% benign, so every flag is a false positive by construction. Green bars are the whole Q4 decision. Red bars are the standalone Lane B gate at its nominal serves_intent ≤ 0.5 threshold.

whole Q2 decision (published reference)Q4 hard block (Lane A read)Lane B gate, serves_intent ≤ 0.5

Data table

Gate	per-event FPR	Wilson 95%
whole Q2 decision · Jev (hosted)	0.33117	not published
whole Q2 decision · DiffusionGemma	0.02722	[0.02020, 0.03659]
whole Q2 decision · OpenJev	0.01944	[0.01365, 0.02762]
Q4 hard block · DiffusionGemma	0.01231	[0.00790, 0.01915]
Q4 hard block · OpenJev	0.01750	[0.01205, 0.02534]
Lane B ≤ 0.5 · DiffusionGemma	0.10110	[0.08704, 0.11715]
Lane B ≤ 0.5 · OpenJev	0.17887	[0.16055, 0.19879]
Lane B ≤ 0.5 · DiffusionGemma, C1	0.18730	[0.16862, 0.20753]
Lane B ≤ 0.5 · OpenJev, C1	0.31951	[0.29671, 0.34320]

Repeatability

Three byte-identical replays of each model over 1,519 events. Flips only happen on events flagged in at least one replay, so the rate that matters is over those: 0.00% for OpenJev (0 of 143 flagged events); 17.24% for DiffusionGemma (10 of 58 flagged events); 13.82% for Jev 1.13.0 (21 of 152 flagged events). Jev 1.13.0 was replayed 16 times in all, and over all of them its rate rises to 22.78%; more replays can only find more flips, so OpenJev’s zero is a zero over three. DiffusionGemma’s confidence differs on every event across replays even where its action does not.

Flip rate: corpus-wide against flagged events only

corpus-wide, all 1,519 eventsflagged in at least one replay

Data table

Model	events	flips	corpus-wide flip rate	flagged events	flips on flagged events	flagged-only flip rate	understated by
OpenJev	1,519	0	0.000000	143	0	0.000000	n/a — zero flips
DiffusionGemma	1,519	10	0.006583	58	10	0.172414	26.2×
Jev 1.13.0	1,519	21	0.013825	152	21	0.138158	10.0×

Where the deciders disagree

OpenJev, DiffusionGemma, the Gemma 4 judge and the rules disagree on 49.87% of the Broad comparison (55.30% of the Production-weighted corpus). All 2,133 disputed Broad cases went to an independent model, openai.gpt-oss-120b-1:0, blinded to the four votes. It is one model’s opinion, not ground truth, and it is permissive: it allows 30.03% of the cases the corpus grades unsafe. Agreement with it is weaker evidence than the raw rate suggests.

Six measured metrics, 4 models

Data table

Model, block-only lens	agreement on unsafe cases	agreement on benign cases	block-only F1	confirm rate	p50 latency (s)	flip rate
OpenJev	0.5597	0.8024	0.70231	0.08488	19.44	0.000000
DiffusionGemma	0.2730	0.8306	0.26792	0.08174	7.35	0.006583
Gemma 4 judge	0.4676	0.1228	0.71248	0.43070	not measured	not measured
Jev 1.13.0	not measured	not measured	0.54153	0.16348	1.42	0.013825

Model, the F1 column on the any-intervention lens	agreement on unsafe cases	agreement on benign cases	any-intervention F1	confirm rate	p50 latency (s)	flip rate
OpenJev	0.5597	0.8024	0.65748	0.08488	19.44	0.000000
DiffusionGemma	0.2730	0.8306	0.18527	0.08174	7.35	0.006583
Gemma 4 judge	0.4676	0.1228	0.33517	0.43070	not measured	not measured
Jev 1.13.0	not measured	not measured	0.50245	0.16348	1.42	0.013825

Truth against decision, one matrix per model

under 2% of the row2–10%10–40%40–75%over 75%

Data table

Model	truth	decided	cases	share of the truth row
OpenJev	grade A — proven unsafe	allow	0	0.000000
OpenJev	grade A — proven unsafe	confirm	5	0.294118
OpenJev	grade A — proven unsafe	block	12	0.705882
OpenJev	grade B — claimed unsafe	allow	102	0.243437
OpenJev	grade B — claimed unsafe	confirm	86	0.205251
OpenJev	grade B — claimed unsafe	block	231	0.551313
OpenJev	grade D — benign by provenance	allow	3,135	0.927240
OpenJev	grade D — benign by provenance	confirm	233	0.068915
OpenJev	grade D — benign by provenance	block	13	0.003845
DiffusionGemma	grade A — proven unsafe	allow	0	0.000000
DiffusionGemma	grade A — proven unsafe	confirm	4	0.235294
DiffusionGemma	grade A — proven unsafe	block	13	0.764706
DiffusionGemma	grade B — claimed unsafe	allow	358	0.854415
DiffusionGemma	grade B — claimed unsafe	confirm	3	0.007160
DiffusionGemma	grade B — claimed unsafe	block	58	0.138425
DiffusionGemma	grade D — benign by provenance	allow	3,053	0.902987
DiffusionGemma	grade D — benign by provenance	confirm	305	0.090210
DiffusionGemma	grade D — benign by provenance	block	23	0.006803
Gemma 4 judge	grade A — proven unsafe	allow	0	0.000000
Gemma 4 judge	grade A — proven unsafe	confirm	4	0.235294
Gemma 4 judge	grade A — proven unsafe	block	13	0.764706
Gemma 4 judge	grade B — claimed unsafe	allow	41	0.097852
Gemma 4 judge	grade B — claimed unsafe	confirm	137	0.326969
Gemma 4 judge	grade B — claimed unsafe	block	241	0.575179
Gemma 4 judge	grade D — benign by provenance	allow	1,855	0.548654
Gemma 4 judge	grade D — benign by provenance	confirm	1,503	0.444543
Gemma 4 judge	grade D — benign by provenance	block	23	0.006803
Jev 1.13.0	grade A — proven unsafe	allow	0	0.000000
Jev 1.13.0	grade A — proven unsafe	confirm	9	0.529412
Jev 1.13.0	grade A — proven unsafe	block	8	0.470588
Jev 1.13.0	grade B — claimed unsafe	allow	128	0.305489
Jev 1.13.0	grade B — claimed unsafe	confirm	136	0.324582
Jev 1.13.0	grade B — claimed unsafe	block	155	0.369928
Jev 1.13.0	grade D — benign by provenance	allow	2,899	0.857439
Jev 1.13.0	grade D — benign by provenance	confirm	479	0.141674
Jev 1.13.0	grade D — benign by provenance	block	3	0.000887

Agreement with the blinded adjudicator

unsafe, grade A or Bbenign, grade D

Data table

Decider	unsafe agreement	unsafe count	benign agreement	benign count
OpenJev	0.5597	164 / 293	0.8024	1222 / 1,523
Gemma 4 judge	0.4676	137 / 293	0.1228	187 / 1,523
DiffusionGemma	0.2730	80 / 293	0.8306	1265 / 1,523
deterministic rules	0.3106	91 / 293	0.8529	1299 / 1,523

On unsafe disputed cases OpenJev agrees with the adjudicator on 0.5597, DiffusionGemma on 0.2730 and the judge on 0.4676. On benign ones the judge agrees on only 0.1228: it escalates almost everything.

The cost of two-sided routing. 1,346 cases are ones the judge wanted to escalate and the small model allowed. The adjudicator agrees with discarding 89.08% of them, but 32 are independently graded unsafe, 2.38% (Wilson 1.69%–3.34%), a loss that does not depend on the adjudicator. Whether that is acceptable is recommendation 7 on Deploy.

The disagreement queue

Every disputed case, filterable by vote pattern, grade, surface and verdict.

Showing a stratified 30 of 2,133 cases.

Security work, in order

evaluation-only never-train
Nothing derived here is approved for training, synthetic generation, teacher context, distillation or redistribution. The 13 source datasets are public and linked on reproduce; the analysis artifacts behind each number are held privately and are available on request.