GitHub (new tab)

Checks

The six checks

How a check works

Every system we can run at a public pin gets one request: 512 prompt tokens, 512 reply tokens, greedy, every reply token checked. An end token does not stop the reply. A system has passed on a model when all six checks passed there.

No check attempts a forged proof. Soundness and weight secrecy are not tested, so the glyph reports what the publisher's documents state about them.

In three of the checks we change one thing the verifier must refuse, and in the other three we test an honest run. A system whose proof is just a hash of the commitment, prompt and reply, computed from the weights the prover loads, would pass all six, and Level 1 would still keep it on the ladder, reading passed. But it would get no position, and what decides that is not a test but its publisher's recorded statement that an accept is the committed model's own greedy reply, or our reading of one.

Run check

The publisher's prover, as published at the pin, ran 3 sessions from a cold start, and every reply got an accepted proof.

passes
The model's prompts are split into 3 slots. Each slot has a complete session: a job from a cold start whose prove call exits 0 with an accepted proof for every request. It is built from the pinned public commit as published there, with no weakening flag.
fails
A slot has 5 incomplete runs, or a recorded refusal by the system's own code with a reason about its inputs, or a counted session used a weakening flag.
cap
A session that reaches the spending cap, 120 USD on Llama 4 Maverick and 40 USD on the others, stops and is incomplete.
counted
A slot counts its first complete session by start time, never one chosen for its figures. A session that missed Verify without weights only for a visible GPU, and was never verified again, gives its slot to the next complete session of the same image and adapter commit that meets the check.

Model check

We change one weight, and no proof made with it is accepted against the original weights' commitment.

passes
Proving with one weight changed, against the commitment to the unchanged weights, gives no accepted proof in 3 of 3 trials of the same change. A prove call that makes no proof counts only when its publisher documents, at the pinned commit, that this outcome means the weights are not the committed ones. An honest proof checked against the changed weights' commitment is refused in 3 of 3 trials of the same pair.
fails
A trial was accepted, and the system's own command, run outside our adapter, accepted the same changed input.

Prompt check

We change one prompt token, and the verifier refuses the proof.

passes
With one prompt token changed, at a position set by a seed and new in each trial, the verifier refuses the proof in 3 of 3 trials.
fails
A trial was accepted, and the system's own command, run outside our adapter, accepted the same changed input.

Reply check

Every proof covers all 512 reply tokens. We change one reply token (the first, the last or one inside), and the verifier refuses the proof.

passes
Every accepted proof covers all 512 reply tokens. With one reply token changed, the verifier refuses the proof at the first position and at the last, 3 of 3 trials of the same change each, and at an interior position set by a seed, new in each of 3 trials.
fails
A proof covered fewer than 512 reply tokens, or a trial was accepted and the system's own command, run outside our adapter, accepted the same changed input.

Verify without weights check

The verifier reached every verdict with no copy of the model and no GPU.

passes
Every counted verdict comes from a verify call with no GPU visible and no file of the model on its filesystem, and the trace of the files it opened names none. A verdict reached with a GPU visible counts when the same proof, verified again in the run's image on a host with no GPU, is accepted under these conditions.
fails
The trace names a file of the model, or a verify call needed one.

Fidelity check

The proved replies match the original's opening tokens better than the bar (a damaged copy of the model) does, by a paired test at p < 0.05.

passes
The replies the counted sessions proved score above the bar, by a paired one-sided sign-flip test at p < 0.05.
fails
The bar scores above the replies, by the same test. With neither shown, the check is not decided.
score
The score of one reply is the number of leading tokens equal to the original's reply to the same prompt, 0 to 512. The original is a Hub copy of the model publisher's release, at the publisher's precision, named with its commit on each system page: 16‑bit for Llama 4 Maverick 17B-128E Instruct and Llama 3.1 8B Instruct, 32‑bit for GPT-2.
bar
The bar is the damaged control with the highest mean score among those that every honest control scores above. The honest controls are the original run again and its 8-bit and 4-bit copies (Q8_0, Q4_K_M). A system's page lists the damaged ones. With none, the model has no bar and the check is not decided on every row.

Checks not decided

A check that is not decided has not failed. So a row that falls short of passing only because checks are not decided reads Not passed yet. A system page can also print a verifier trial from outside the counted suite, beside the check whose change it makes, but that trial sets no state.
system limit
The system cannot do what this test needs (its publisher's words are quoted), so the row cannot pass.
not run
Not run yet.
not resolved
The test ran, and its result does not count yet.
not reproduced
A verifier accepted a changed input in our adapter, and we have not yet repeated it with the system's own command.

What we test and what we read

WhatWhereHow we know it
completeness: the publisher's prover, run honestly, gives accepted proofsRunwe test it
binding: a changed weight, commitment, prompt token or reply token is refusedModel, Prompt, Replywe test it
verdicts reached with no model file and no GPUVerify without weightswe test it
how close the proved replies stay to the original'sFidelity check, Fidelity petalwe test it
every figureCost vs serving, Tokens per second, Cost per reply, Verify time per reply, Proof size per reply, Fidelity figure, KL to the original weightswe test it
soundness, as a stated boundIntegritywe read it from the publisher's documents
zero knowledgeConfidentialitywe read it from the publisher's documents
formal proofsFormally verifiedwe read it from the publisher's documents
setupTrust assumptionswe read it from the publisher's documents
the stated security levelOrderwe read it from the publisher's documents
that an accept is the committed model's own greedy replyOrderwe read it from the publisher's documents
soundness against a cheating proverIntegrityno check tests it, and the petal prints what the publisher states
zero knowledge of the programs the row runsConfidentialityno check tests it, and the petal prints what the publisher states

Figures

Cost vs serving

multiple
m over b, on one model. GPU time counts only the H200 GPUs bound to the job.
m
GPU-seconds per proved reply, generation included: the mean of 3 sessions. Proofs released within 1 s of the previous one form a release group. m runs from the first group's last release to the session's last, over the replies released after it. Where a system generates all its replies in a phase of its own, m adds that phase's GPU-seconds over the session's replies.
b
GPU-seconds per reply when the same request is served without a proof. The model publisher's own release (FP8 on Llama 4 Maverick) runs on the pinned engine and the fewest NVIDIA H200 GPUs that hold it, at full load. Full load is the lowest concurrency, doubling from 1 to 1,024, whose rate reaches 95 % of the highest. If that is 1,024, the model has no b (not saturated).
whole sessions
A row has m when each of its 3 sessions has at least 3 groups after the first, each group's time per reply within 5 % of their median. Otherwise it has no multiple and prints its session GPU-seconds per reply over b, labelled “whole sessions of N replies”.
parts
Where a system generates its replies in a phase of its own, the figure splits in two, each over b: generation, and the rest (proving under m, commit and proving under whole sessions). Not every system times them apart, so they print only under Cost on its own page. The order reads the figure, never a part.

Tokens per second

The request's 512 reply tokens over the GPU-seconds one reply took, on one GPU, printed in tokens/s per GPU with the GPU named. The proved rate counts whole sessions, as Cost vs serving counts them. The served rate is for the same reply served without a proof, at full load, on the same GPU type. Served over proved is Cost vs serving.
Both depend on the GPU, so neither ranks a row: the order reads m, the steady multiple (OR-3), never a rate.
Prompt side (prefill) and reply side (decode) are not timed apart in the records.

Cost per reply

steady
m in USD, at the release's price for one H200 GPU-hour.
session
GPU-seconds from the start of commit to the release of the session's last proof, over its replies, generation included, in USD at the same price: the mean of 3 sessions.

Verify time per reply

The wall time of the verify calls on a session's proofs, summed, over the session's replies: the mean of 3 sessions. A second verify never replaces the session's own time.

Proof size per reply

The bytes of every file of a session's proofs, as stored, over the session's replies. A proof that covers several replies is counted once. The figure is the mean of 3 sessions, in MB (1 MB is 1,000,000 bytes).

Fidelity figure

The share of reply positions where the system's top token is the original's reply token, when the system is fed the prompt and then that reply token by token. It is the mean over the prompts, as a percentage, with 0 for a prompt the system's own code declines. The model's 8-bit and 4-bit copies' percentages (Q8_0, Q4_K_M) are printed under it.
100 % is the original's choice at every position. It counts chosen tokens and compares no probabilities. KL compares them.
Computed by the system's own code on the reference reply, not by the proved computation. It sets no state.
Why top-token agreement: a greedy reply is the top token at each position, so this counts how often the proved model makes the original's choice, at every one of the 512 positions, not only up to the first difference.
Where a proved reply starts like the original's, both models have read the same text, so a proof states which token the proved model picks there, up to and including the first token that differs. The system's code must pick that token at every such position; the system's page prints the count. A row whose code picks another token there prints the Fidelity check's state in the figure's place.

KL to the original weights

Mean KL divergence per scored token, in nats, of the file the system commits against the original, over 64 texts of 512 tokens. Computed by llama.cpp on the committed file, not the proved computation. We print it, but it gates nothing.
not measured
Printed with the reason where llama.cpp cannot run the original (Llama 4 Maverick's 16-bit file), or cannot load the committed file and the system's adapter returns no distributions of its own.

Order

ranked
Only systems that passed on a model are ranked.
unranked
A system that passed is ranked only if its publisher states, at the pinned commit, that an accept means the reply is the committed model's own greedy reply to the prompt, or is read as stating it (marked read as, with its quotes), and the level file records a numeric security level that its publisher states there, for at least one verifier it ships. Otherwise it is listed as Passed, not ranked, with the reason. Level 1 sets no minimum level.
above
One system is placed above another when both have a multiple, its mean multiple is at least 10 % lower, and its highest session m is below the other's lowest. A system with a multiple is placed above one without. Two systems without one compare the same way, on session cost per reply.
position
A system's position is 1 plus the number of systems placed above it, so equal counts share a position. It is printed beside the count of systems run on that model.
below
Under the ranked systems come the systems that passed and are unranked, then those that have not passed, each group in alphabetical order. A system that has not passed prints its checks and figures and has no position.
largest model with a passed row
The largest model, by total parameters, with at least one system that passed and whose publisher states that an accept is the model's own reply.
glyph
The glyph never moves a position.
models
Nothing is ranked across models.

Glyph

The glyph shows how close a row is to a perfect prover, one petal per property, and the petals are never added up. We grade four of them on the publisher's documents at the pinned commit. We grade the fifth, Fidelity, on our fidelity run, and it is not decided whenever the Fidelity check is. A failed check sets Failed on the petals whose claim it contradicts, a check not decided lowers none, and a passed check raises none. On any doubt, the lower step wins.

Met Mostly met Partly met Not met Failed Not stated Not decided

Each system page prints what each petal misses for Met, and who acts, computed from this table.

Integrity

soundness

Can a provider get a reply accepted that the committed model did not compute? We grade Integrity on the chance of that, as the publisher states it, and on whether the statement covers what we run. Integrity protects the customer. Met: a proved chance of at most 2^-128 (128 bits) that covers what we run.
protects
the customer
question
The bound the publisher states on a false reply being accepted: whether it is proved, and whether it covers what we run.
Met
A proved bound that covers what we run, every exponent 128 or more.
Mostly met
The same with every exponent 100 or more, or Met resting on our reading.
Partly met
Any other stated bound above Not met: conjectured, not shown to cover what we run, or an exponent from 80 to 99 on a proved bound.
Not met
The bound holds only against a weaker prover, has an exponent under 80, or has an exponent under 100 and is not proved.
Not stated
The documents read at the pinned commit give no bound with its exponents.
Failed
Our Model, Prompt or Reply check failed.
Not decided
We have not read the publisher's documents at the commit we run.
covers what we run
The bound holds for the settings we run, against any prover, per reply (or per proof with one reply a proof), with each reply checked alone. The publisher's or our records show its conditions hold for the files we run, and the verifier program we run is proved or stated to implement what the bound is about. A batch size not recorded is not covered.
exponent
In 2^-t, the exponent is t, and a larger exponent is a smaller chance. An error term q/2^t also has exponent t. A bound that is a sum of terms needs every term's exponent, so the whole stays within a few times the smallest chance allowed. We read exponents as the publisher states them, compute no bits figure and fix no number of hash queries q. We read a bound stated per part of a proof (a segment, a layer) per part, as stated. Such a bound is not per reply, which is a gap in covering what we run, and that gap lowers the step no further.
Integrity does not grade completeness. It grades soundness only, and completeness is what the Run check tests. Our Model, Prompt and Reply checks can set Failed, but they do not measure the bound.

Confidentiality

zero knowledge of the weights

Does checking a proof reveal the model's weights? We grade Confidentiality on what the publisher states and proves about keeping them secret. Confidentiality protects the provider. Met: zero knowledge proved for what we run.
protects
the provider
question
What the publisher states and proves about a proof showing nothing of the weights' values.
Met
A zero-knowledge theorem proved for the configuration we run, batch size included, and stated for its programs. A published list of what stays visible, with nothing on it that depends on the weights.
Mostly met
A zero-knowledge theorem is proved and a condition of Met is missing.
Partly met
The publisher states that the weights stay hidden and proves no zero-knowledge theorem.
Not met
The verifier is given the weights, or the publisher states that they are not hidden.
Not stated
The documents read say nothing on hiding the weights.
Failed
Our Verify without weights check failed.
Not decided
We have not read the publisher's documents at the commit we run.
Confidentiality does not grade hiding the architecture. Level 1 models are open-weights, so their architecture is public, and a closed-weights level would grade it. Our Verify without weights check can set Failed, but it does not test what a proof reveals.

Fidelity

quantization error, in matched leading tokens

Is the model that is proved as good as the one its publisher released? Proving usually runs a copy whose numbers are stored in fewer bits (quantization). For Fidelity, we compare the proved copy's replies with the original's, using our 8-bit and 4-bit copies as yardsticks. Fidelity protects the customer. Met: shown closer to the original run again than to the 8-bit copy.
protects
the customer
question
How close the proved replies stay to the model publisher's own release, against the model's 8-bit and 4-bit copies.
Met
Replies shown above halfway between the original run again and the 8-bit copy, in a fidelity run of one request at a time.
Mostly met
Replies shown above halfway between the 8-bit and the 4-bit copy.
Partly met
  • The Fidelity check passed, and the replies are not shown above halfway between the 8-bit and 4-bit copies.
  • Or no claim card is found, or its policy accepts tokens other than the model's top token.
  • Or a copy a halfway point needs did not run, failed, or was scored on another number of replies.
Failed
The Fidelity check failed: the bar (a damaged copy) scores above the replies.
Not decided
The Fidelity check is not decided, or no fidelity record is stored.
halfway
The average of two copies' mean scores, from the unrounded means.
one request at a time
Every part of the rung's fidelity run sends one request at a time in llama.cpp (slots 1). Where llama.cpp at the level's pin cannot run the original, the original and its second run go to vLLM, which takes every prompt in one call, so that part does not count as one at a time. There the second run sends the prompts in reverse order, so its batches differ from the first run's.
shown above
The Fidelity check's paired sign-flip test gives p < 0.05 for the system's scores against the halfway point.
measure
We grade the petal on matched leading tokens, the measure the Fidelity check uses. The Fidelity figure is top-token agreement and sets nothing.

Formally verified

machine-checked soundness proof

Has a computer checked the mathematical proof that false replies are refused, and does that check reach the code we run? It protects the customer. Met: the verifier we run is machine-checked.
protects
the customer
question
How far a machine-checked proof of soundness reaches: part of the protocol, an abstraction of it, the protocol with the model, or the verifier code the row runs.
Met
Built: the verifier we run, from the bytes it reads to its verdict, with its trusted base listed and no input-reading code outside the proof.
Mostly met
Built: the protocol, model included. Or the verifier we run, with a gap in its trusted base or its input-reading code.
Partly met
Built: an abstraction of the protocol (the protocol without the model, or the model without the protocol), or part of the protocol. A theorem whose conclusion is not about the model, or a build with axioms beyond its tool's standard ones, also stops here.
Not met
The publisher states that nothing is machine-checked.
Not stated
The documents read name no machine-checked proof.
Failed
Our build of the named theorem failed or reported axioms other than the declared ones. Or our Model, Prompt or Reply check failed where the stated step is Met.
Not decided
A machine-checked proof is stated, and our build of it is not recorded yet, or the recorded one stopped before a verdict. Or we have not read the publisher's documents at the commit we run.
build
Each step above Not met needs a matching build of the named theorem: one recorded at the pinned commit with the publisher's toolchain. We run the publisher's own command for it, unpatched, and record what it reports. We do not review what the theorem states. We read the step from the publisher's words, printed with the petal.
above Partly met
A step above Partly met needs a theorem whose conclusion is about a reference definition of the model, and a build that reports only its tool's standard axioms (for Lean, propext, Classical.choice and Quot.sound), so no sorry. The system page says that no outside reader of the proof has reviewed the statement.
the code
Only the verifier we run counts as the code. A verified verifier we do not run counts at the protocol level.
protocol or code
A proof about a formal model of the method, rather than about a program, verifies the protocol, not the code. Level 1 says the protocol, model included, when that formal model includes this model's computation, and an abstraction of the protocol when it leaves the computation out (or covers the model without the method). Only a proof about the verifier program we run verifies the code.
scope card
We print it under a system's petals, and it sets no state, since the petal grades soundness only. Soundness is the verifier's and zero knowledge the prover's, so the cells that ask them of the other party do not apply.
Formally verified does not grade prover code, which bears on completeness and zero knowledge. The scope card prints what is verified for it, and the card sets no step. Limits a publisher states beyond the five things graded here (reach, conclusion, input-reading code, axioms, our build) do not lower the step. The rules leave two questions open: whether a step above Partly met should need a record that the verifier we run accepts honest proofs at our files, and whether our trial of another of the publisher's verifiers should set a step. Today such a trial sets none.

Trust assumptions

trusted setup or party

Does checking a proof rely on anyone's honesty besides the mathematics, such as whoever ran a one-time setup or holds a hardware maker's key? It protects the customer. Met: nothing to trust but cryptography, as the publisher states.
protects
the customer
question
What checking a proof relies on besides cryptography: a setup secret, a fixed set of parties, or a hardware maker's or another trusted key.
Met
The publisher states that soundness rests on cryptography alone: no setup made a secret. Met also needs proof files anyone can check, unless the publisher is silent on who can check them.
Mostly met
An open setup ceremony with a public transcript, safe if one participant was honest. Or Met's facts are our reading, not stated in words.
Partly met
One honest party among a fixed set, a challenger within a window, or proofs only a designated verifier can check.
Not met
A setup run by one party, a hardware maker's key or another trusted key.
Not stated
The documents read say nothing on what soundness rests on.
Not decided
We have not read the publisher's documents at the commit we run.
This petal does not grade which model a commitment names. Nor does it grade how strong soundness is (a theorem or a conjecture), which Integrity grades. Where a publisher's theorem asks that the public files were fixed before the hash function was drawn, Integrity counts that condition.

Why a petal is low or not decided

not stated
Not stated (drawn as an outline): the publisher's documents read at the pinned commit are silent on the property. Nobody can rely on a guarantee nobody states, so we grade it as none, never as not decided.
below Partly met, as the publisher states it
Not met or Failed (drawn filled, name struck): the publisher states that the property does not hold, or what it states falls below Partly met.
a check failed, or the proof build failed
Not met or Failed (drawn filled, name struck): a check failed, or the matching build of the publisher's machine-checked proof failed or reported axioms other than the declared ones.
not read yet
Not decided (drawn hollow): our record of the publisher's documents is missing, in another form or read at a commit other than the one the row runs, or the row has no recorded commit. We cannot call documents silent before we have read them, so the petal is not decided, never Not stated.
Fidelity check not decided or not recorded
Not decided (drawn hollow), Fidelity only: the Fidelity check is not decided, or the row has no fidelity record or no scores.
waits on our finished build of the publisher's machine-checked proof
Not decided (drawn hollow), Formally verified only: the publisher states a machine-checked proof, and no build of the named theorem is recorded at the pinned commit with the publisher's toolchain, or the one recorded was stopped at its time limit or by a kill before it reached a verdict. Where the publisher names no toolchain, no build can match.

Our reading

At least one fact behind the petal's step is our reading, not the publisher's words. The petal says it comes from our reading, the publisher's quotes are printed with it, and it is graded Mostly met at most, for every system alike. We never read the name of a construction as a claim.

Systems not run

We list a system with no row after the ladder, with one reason: the first of these that holds in its sources, in this order. Each system's line is the fact that sets its reason, linked to where we read it.

Licence

No licence file, or a licence that grants no use. We read a licence that grants internal use for testing and evaluating as covering runs we make ourselves, but it grants no right to hand the software to anyone else.

Not yet run

Nothing above holds, and no session has run yet.