GitHub (new tab)

Learn · For prover teams and engineersGlossary

How we compare

Every system gets the same request. Those passing the six checks, with a stated security level, are ranked by cost vs serving inside one model.

1 of 3

Every system runs on H200 GPUs at one price. The harness times each system from outside, not with its own code (new tab), and we check every system against the same statement. No two runnable systems share a model at their pinned commits (RFC-0018 (new tab)), so each model has its own table. Nothing is ranked across tables, because Maverick (a 400B mixture-of-experts) and a dense 8B differ in what serving costs and in how proving scales.

Fixed before the runs

We froze the level file before the first counted run, and every counted run names its digest. The prompts are 512-token windows of Project Gutenberg books. They are cut with a seed from a public randomness beacon (new tab), at the beacon's first round after the time of the freezing commit. That time comes from the committer's own clock, and GitHub recorded each freeze 2 s after it. We froze the level three times on 2026-11-11, and each freeze drew new prompts (PR-2 (new tab)). Every counted run uses the third draw.

We run each system at a full commit hash, and the adapter uses no undocumented setting. FlashProve runs its upstream commit unchanged. SASH runs a commit of a pull request that this project's owner wrote and upstream has not merged (ties).

The request is 512 prompt tokens and 512 reply tokens. It fills FlashProve's 1,024-token window and uses the one block size in SASH's own recorded runs (RFC-0017 (new tab)).

The checks

Timing an honest run cannot show that a verifier refuses a changed model, prompt or reply. So six checks gate a position. They run the publisher's own prover honestly, which makes them sanity checks, like a passing CI run: honest runs verify, and the changes we make are refused. But they do not show that a cheating prover gets caught. We verify each changed input beside its honest twin (new tab), and a trial whose twin is not accepted is void.

The Fidelity check is a floor against gross damage. On the 8B model it rests on 15 replies, so it is coarse.

The figure

From the board, release v1.0, runs to 2026-11-16

2 rows so far: 2 passed all six checks.

On Llama 4 Maverick 17B-128E Instruct
SystemStandingCost vs servingVerify timeProof size
FlashProvePassed1,317×3 whole sessions of 120 replies17.83s/reply18.78MB/reply
On Llama 3.1 8B Instruct
SystemStandingCost vs servingVerify timeProof size
SASHPassed62,160×3 whole sessions of 5 replies1,078s/reply3,909MB/reply
A row prints whole sessions when any of its three sessions has too few or too uneven release groups.

An exact-match proof covers only the committed copy's own greedy reply, so the system first writes every reply with that copy. Cost vs serving counts that generation in both forms, m over b and whole sessions over b.

commitgenerationgroup 1group 2group 3group 4generation: counted in msteady interval: the GPU time behind mwhole session: from the start of the commit stepcommitgenerationgroup 1group 2group 3group 4generation:counted in msteady interval:the GPU time behind mwhole session: from the startof the commit step

FlashProve's replies come from its integer-model package (new tab), which runs “greedy decoding, one sequence at a time, one GPU”. We serve the baseline at full load, since serving one request at a time would make every multiple smaller.

From the board, release v1.0, runs to 2026-11-16

Llama 4 Maverick 17B-128E Instruct

Serving Meta's FP8 release, vLLM 0.32.0, 4 × H200, full load at 512 concurrent requests · n = 3 timed batches, 1 run: 0.1918GPU-s/reply

Llama 3.1 8B Instruct

Serving Meta's BF16 release, vLLM 0.32.0, 1 × H200, full load at 256 concurrent requests · n = 3 timed batches, 1 run: 0.04311GPU-s/reply

GPT-2

Serving not run yet

The serving engine runs at its defaults, not at its highest throughput. It serves the model publisher's release, and provers run quantized copies, so the multiple includes that difference in precision. The price covers only bound GPU time, idle time included, and not CPU, host memory, disk or network.

Order and the glyph

Level 1 sets no floor under the numeric security level a position needs, because tests cannot tell 60 bits from 128 and stated levels are not comparable without recomputing them.

On the glyph, when a publisher says nothing about a property, we grade its petal Not stated, because nobody can rely on a guarantee nobody states. A passed check never raises a petal above what the publisher states, since a refused change shows only that one attack was refused. The glyph never moves a position.

Judgement and changes

These values were set by judgement before the runs:

  • the 10 % margin of the order rule
  • the 5 % steady and full-load tolerances
  • three trials per changed input
  • p < 0.05 in the fidelity test

The Integrity petal's cut points were adopted on 2026-11-13, after the freeze. The 128 and 100 come from the Ethereum Foundation's zkEVM security milestones (new tab), and 80 is a strength that NIST SP 800-57 Part 1 Rev. 5 (new tab) disallows. Its Table 4 disallows every strength under 112 bits for applying protection and accepts 112 only through 2030, so the 80 and 100 cuts sit below NIST's floor.

Some rules added to RFC-0018 (new tab) after the last freeze decided rows. Under one of them, FlashProve's Model check passed: given changed weights, its prover answered “failed” with no proof, as it documents (new tab) (CH-1a). Under another, two SASH slots went to their next complete sessions, because the replaced ones verified with a GPU visible and kept no proofs to verify again (SE-3a).

Getting a row

A row is one system on one model of the ladder, and the level chooses those models, never an entrant (LV-2 (new tab)). We run every row's three counted sessions ourselves, so a run you make elsewhere can go in the issue, but it is not a counted session.

To get a system on the ladder, open an issue (new tab) naming the upstream repository, the full commit hash and the workloads the system supports. The adapter then lands by pull request (Writing an adapter (new tab)). A system with another statement or another kind of check is listed after the ladder as not run.

When a changed input is accepted in our adapter, the check does not fail yet. It fails only once the system's own command, run outside the adapter, accepts that input too. Until then the check reads not decided, and we tell the publisher under embargo (new tab).