Reproduce

Every figure comes from a JSON or text artifact written by a named script, over a prediction file with a recorded hash, at a fixed seed. The analysis artifacts are held privately and are available on request; the source datasets are public.

Sources

The 13 sources that supply rows are public and reachable without a token. Each is download-only: the table links them and this site republishes none of their rows. Licences in use: Apache-2.0, CC-BY-4.0, CC-BY-NC-4.0, MIT. coding-agent security benchmark is CC-BY-NC-4.0 and its 279 rows must not be used for any commercial purpose.

9 of the sources are Hugging Face datasets and can be previewed below in the Hugging Face viewer, with the pinned revision beside each; the other 4 are on GitHub. Every preview is collapsed and loads nothing from huggingface.co until opened, and the rows it shows are served by Hugging Face from the source repository, not republished here.

What is withheld, and the one carve-out

Code paths

Everything below is in feat/system-one-benchmarks, and pinned at d2ae73f32736, which is the tree these figures were produced from. The default branch does not carry this work: of the 11 paths below, only benchmarks/datasets.lock.json resolves on main, and there it is a different file. Each row links the branch copy and the pinned copy; a branch moves.

Seeds and model revisions

Von 1.0.1 would not answer under the library versions its SDK declared (80 of 80 requests failed with HTTP 422), so the environment was upgraded (transformers 4.57.6 → 5.17.0) and the artifact left as pinned.

Settled-file rules

1. Gate on completion and hash. A prediction file is read only if its .meta.json says complete: true and its on-disk SHA-256 matches the recorded prediction_sha256, or if it is a recorded, byte-reproduced merge of files that each pass that rule.
2. Reproduce before extending. Every analysis re-derives at least one published number through its own code path before emitting a new one.
3. New paths only. No existing scorecard, prediction file, manifest or scorer was modified by any analysis.
4. Publish the integrity counters: row counts, error rows, duplicate (case_id, event_index) pairs, rows missing probabilities and the case-set intersection between arms.

Worked example: the intent reversal

1. Take the corpus and the eight prediction arms from outputs/intent-real/. Check cases.jsonl hashes to 85dfc730d1337bb3… and that all eight manifests report one cases_sha256.
2. Per arm, verify complete: true and the on-disk SHA-256. Abort if either fails.
3. Run benchmarks/scripts/score_intent_separation.py unmodified, with --lens both --lane both --per-event --bootstrap 2000 --seed 741983 --question Q2 --context {C0|C7}.
4. Confirm the control: the same script, flags and seed on AgentDojo give -0.0208 (C0) and -0.1246 (C7) for Jev 1.13.0 on the lead cell (grade-A lane, block-only, per case, sep_vs_resisted). If the control does not reproduce the old negative, stop.
5. Read the lead cell on the proof-backed corpus. Expect +0.2667 for OpenJev at C0 and +0.5750 at C7.
6. The within-family contrast covers 325 families; all 32 cells should be positive with intervals excluding zero.

Site build and gates

• 1152 assertion checks, each a (file, key path, expected value) triple. A mismatch prints and writes nothing.
• Reading precision. Every figure is published rounded, never above six significant digits and money to cents; the artifact’s exact decimal is kept in the page (hover it, or use Exact values in the header). The verifier fails a page with any longer visible number or any exact value that does not round-trip.
• Structure and links. Every internal link must resolve to a page and an anchor; unresolved template tokens, overflowing chart labels and a tooltip inside a heading all abort the build.
• Payload guard. Before upload every file is scanned against the corpora themselves for case ids and verbatim text, plus CJK and credential patterns; a hit aborts.
• No external assets. Inline SVG, one inlined stylesheet, inline script the pages work without; the only embeds are the collapsed Hugging Face viewers above.

The build writes _build-figures.json beside the pages: every substituted value, the artifacts read and the assertion count.

Experiment ledger

Restated figures

Where an earlier figure disagreed with the artifact, the artifact is the record.

evaluation-only never-train
Nothing derived here is approved for training, synthetic generation, teacher context, distillation or redistribution. The 13 source datasets are public and linked on reproduce; the analysis artifacts behind each number are held privately and are available on request.

Upstream defects reported: vllm-project/vllm#58015, wfzyx/von#11.