GitHub (new tab)
Standing
Passed
Model
Llama 3.1 8B InstructDense, 8.03B
Version
PR #36 (new tab) · not merged upstream · by this project's owner · upstream eb95bfa: SASH's own code refused one Level 1 prompt (record (new tab))

Glyph

GradesMetMostly metPartly metNot metNot statedFailedNot decided

Petal, grade, what is missing first, what Met needs and what it has
IntegrityCan a fake reply pass the check?Partly met
The limit names no verifier.Met needsa proved limit of 2−128 for boundary-grid --verify, which we runHasabout 2−100 per segment, conjectured7 missing · publisher acts, Integrity
Missing

For Mostly met

  1. A limit for boundary-grid --verify
  2. A theorem, not a conjecture
  3. A statement that it holds against any cheating prover
  4. A limit per whole reply
  5. A record its conditions hold for our files

For Met

  1. Every part of the limit at most 2−128
  2. The publisher's words for each point we read
Evidence
Publisher's words at 4f41667 (3)“stwo's own conjectured-soundness accounting is security_bits = pow_bits + log_blowup_factor * n_queries so blowup 2 (log_blowup_factor = 1) at 100 conjectured bits means exactly 100 FRI queries and no grinding.” (new tab)“A proof made by this prover is about one segment: one transformer layer” (new tab)“assert_eq!(cfg.security_bits(), 100, ‘security parameter drifted’);” (new tab)
Our notes (1)

The security parameters are constants (util.rs L14-L16), and pcs_config() asserts 100 bits (L26). The row runs the sound defaults.

It does not use our reading that the limit is for the settings we run. Met needs the publisher's words there too.

Our Model, Prompt, Reply checks: passed ↓

Why this grade

Partly met: the limit is not shown to cover what we run (items 1 to 5 under Missing). It is not a theorem, and its exponent is 100.

Met needs, in full

A proved limit of 2−128 on a fake reply passing the verifier we run.

What it has

The publisher states 100 conjectured bits of security per segment of a proof: a chance of about 2−100 that a fake segment passes. The figure comes from the accounting of stwo, a proof library. Conjectured means it rests on an unproven assumption, not a theorem.

One segment proves one transformer layer (“A proof made by this prover is about one segment: one transformer layer” (new tab)). A reply passes through every layer of the model, but the publisher states no limit for a whole reply.

The publisher names no verifier for this figure, and does not say which cheating provers it holds against. We run boundary-grid --verify, the publisher's verifier command. The publisher states that this command runs the publisher's check, verify_composed (“verify_composed – the only accept path, and what boundary-grid --verify runs on every chunk” (new tab)). The quotes we hold do not say how a chunk relates to a segment.

Each item today
  1. Today no verifier is named.
  2. Today the 100 bits rest on an unproven assumption.
  3. Today the documents we read do not say.
  4. Today it is for one segment (one transformer layer).
  5. Today the documents we read name none and show none.
  6. Today it is about 2−100 (100 bits).
  7. Today this is our reading.
Our records

In each of our Model, Prompt and Reply tests, we give the verifier a proof with the wrong model, prompt or reply. Each was refused on this run.

In the field's terms

Soundness. Bound: 100 conjectured bits per segment (one transformer layer). Basis: conjectured, stwo's accounting. Stated for: no verifier named. We run: boundary-grid --verify (no bound claimed for it). Adversary: not said. Unit: segment. Batch: not said. Ours is 1 reply. Parameters: row (our reading). Conditions shown for our files: no.

ConfidentialityDoes a proof keep the weights secret?Not stated
We read 4 documents and found nothing on hiding the weights.Met needsa zero-knowledge theorem for exactly what we runHasthe verifier gets no weights, as stated5 missing · publisher acts, Confidentiality
Missing

For Partly met

  1. A statement that a proof keeps the weights secret

For Mostly met

  1. A zero-knowledge theorem

For Met

  1. The theorem for what we run
  2. A statement that its programs keep the weights secret
  3. A published list of what stays visible
Why this grade
Met needs, in full

A theorem that proofs of exactly what we run reveal nothing about the weights.

What it has

The 4 documents we read at the version we ran (4f41667) do not say whether a proof reveals the weights. They state that the verifier does not hold the weights (“The model root is not derivable – it is the publisher's commitment to weights the verifier does not hold” (new tab)), but that does not mean a proof hides them.

Our records

In our Verify without weights test, we check that every verdict came with no model file and no GPU. It passed on this run.

In the field's terms

Zero knowledge of the weights. Stated: not stated. Proved: no. Programs: not stated. Weights given to the verifier: no. Visible list: none published.

FidelityDoes the proved model answer like the original?Partly met
Our test does not show an average above 55.7 tokens, which Mostly met needs.Met needsan average shown above 309.0 tokens, one request at a timeHas31.00 matched tokens (8-bit copy 106.1, 4-bit copy 5.267)2 missing · publisher acts, Fidelity
Missing

For Mostly met

  1. An average shown above 55.7 tokens

For Met

  1. An average shown above 309.0 tokens
Why this grade
Met needs, in full

An average above 309.0 matched tokens, shown by our test, in a run of one request at a time.

What it has

We sent 15 prompts to SASH and to the original model (Llama 3.1 8B Instruct, 16-bit). On average, SASH's replies match the original's for their first 31.00 tokens, up to the first difference (512 at most).

For scale, we scored other runs on the same prompts: the original model run a second time scores 512.0 (every token, since 512 is the most a reply has), our 8-bit copy 106.1, our 4-bit copy 5.267, and a copy we damaged on purpose (3-bit) 1.400, the pass bar every system must beat.

The publisher states that SASH proves a copy with 8-bit integer weights for part of its weights, and states no width for the rest. Values between its layers are floating-point numbers. SASH's copy and our 8-bit copy (Q8_0, a llama.cpp format) are different 8-bit copies. Our test measures the gap, not its cause.

Our run sent one request at a time, as Met needs.

Each item today
  1. Today the average is 31.00 tokens, not shown above it. The number format of the copy SASH proves is the publisher's choice, so this item is the publisher's.
  2. Today the average is 31.00 tokens, not shown above it.
Our records

Our fidelity run on Llama 3.1 8B Instruct, 15 prompts, each reply scored against the original's.

In the field's terms

Quantization error, in matched leading tokens (mean, n = 15). Above T_upper, 55.7 (Mostly met): not shown. Above T_green, 309.0 (Met): not shown. Test: paired sign-flip per prompt (exact), alpha 0.05. Proved: 8-bit integer weights, floating-point activations, for part of the weights. Decoding: exact match (only the model's top token). Requests at a time (slots): 1 for the whole run (llama.cpp).

Formally verifiedHas a computer checked that fake replies are refused?Not stated
We found no computer-checked theorem named in the 4 documents we read.Met needsa rebuilt theorem reaching boundary-grid --verify, which we runHasnothing stated4 missing · publisher 3, us 1, Formally verified
Missing

For Partly met

  1. A named computer-checked theorempublisher
  2. Our rebuild of itus

For Mostly met

  1. A theorem of the whole method, the model's computation includedpublisher

For Met

  1. A theorem reaching boundary-grid --verify, which we runpublisher
Why this grade
Met needs, in full

A computer-checked theorem that the verifier we run refuses fake replies, rebuilt by us.

What it has

The 4 documents we read at the version we ran (4f41667) name no computer-checked theorem.

Our records

Our Model, Prompt and Reply tests passed on this run.

In the field's terms

Machine-checked soundness proof. Tool, theorem and trusted base: not stated. Scope: the scope table further down this page ↓

Trust assumptionsDoes checking trust a setup or a party?Mostly metour reading
No single sentence of the publisher's says it.Met needsthe publisher's words: cryptography only, no setup secretHascryptography only, no setup secret, on our reading1 missing · publisher acts, Trust assumptions
Missing

For Met

  1. The publisher's own sentence saying so
Evidence

From the publisher's words and our reading

Publisher's words at 4f41667 (6)“stwo's own conjectured-soundness accounting is security_bits = pow_bits + log_blowup_factor * n_queries so blowup 2 (log_blowup_factor = 1) at 100 conjectured bits means exactly 100 FRI queries and no grinding.” (new tab)“The preprocessed trace is two committed trees (a shape tree whose root the verifier derives from the public parameters, and a model tree whose root is compared with the published model commitment).” (new tab)“They are committed as a tree of their own, and what says they are the columns the public parameters give is that the verifier builds that tree itself and compares its root with the root the statement declares.” (new tab)“security (the FRI parameters and the hash)” (new tab)“Fixed — produced at setup, trusted by the verifier: the shape and gadget parameters that determine every column size and selector, the security config, the tree column sizes, and the pinned preprocessed root (every table, selector, weight and scale column lives in tree 0, and B4 §3 / B5 / B6 all showed a corrupted table is invisible to everything else).” (new tab)“The verifier (verify_composed) takes those two byte strings and the serialized proof, and nothing else” (new tab)
Our notes (1)

We apply the test FlashProve's row meets: we read a petal as transparent when a sentence among its quotes says what soundness rests on and none of it is a setup that produced a secret (GL-10, amended 2026-11-15). SASH states its soundness as stwo's conjectured accounting over the FRI parameters (util.rs L9-L12). It says the setup's two trees are a shape tree the verifier derives from the public parameters and a model tree compared with the published model commitment (prover/README.md L117-L119, docs/statement.md L127-L129). The Fixed statement it calls trusted by the verifier is those parameters and roots (compose.rs L7-L11). No sentence says in words that no setup produced a secret, so we read the petal as transparent and grade it Mostly met at most, as FlashProve's is.

No test of ours bears on it

Why this grade
Met needs, in full

The publisher's own sentence that checking needs no secret setup and no trusted party.

What it has

The publisher states that the verifier takes two byte strings and the proof, and nothing else. We read this sentence to mean that anyone can check a proof, but the publisher does not say so in words. (the publisher's words: “The verifier (verify_composed) takes those two byte strings and the serialized proof, and nothing else” (new tab))

The publisher states its soundness as stwo's conjectured estimate (see Integrity). Its setup commits two trees: a shape tree, whose root the verifier builds itself from the public settings, and a model tree, whose root the verifier compares with the published commitment. It calls some values Fixed, “produced at setup, trusted by the verifier”. It lists them as the model's shape and the settings of its circuits, the security settings, the trees' column sizes, and the pinned root of the tree that holds every table, weight and scale column. We read “trusted” here to mean that the verifier takes these values as given. These are settings and a tree root, not the weights themselves, and the publisher calls none of them a secret.

So, on our reading, soundness rests on cryptographic assumptions only, and no setup step produced a secret. But the publisher does not say this in one sentence.

Each item today
  1. Today this is our reading.
In the field's terms

Trusted setup or party. Rests on: transparent (no setup made a secret) (our reading). Publicly checkable: yes.

Completeness: 15 of 15 replies accepted, our Run check

Stated security, scope card

What is verified, and for which code, as the publisher states it. It sets no state.

Scope card
WhatBound, as statedProtocolThe verifier we runThe prover we run
Soundness100 conjectured bits (new tab)not statednot stateddoes not apply
Zero knowledgenot statednot stateddoes not applynot stated
Completenessnot statednot statednot statednot stated

Statement

Verifierboundary-grid --verifyclaim card (new tab)
An accept meansthe committed model's own greedy reply to the prompt (our reading: no single sentence states it)sources 1 (new tab) 2 (new tab) 3 (new tab) 4 (new tab) 5 (new tab)
From our commit recordthe commitment, 2 filessources 1 (new tab) 2 (new tab)
From the requestthe prompt (512 token ids)source (new tab)
From the proof199 files, and the verifier reads the reply from themThe proof's Fixed statement carries the gadget switches, so a proof made with an --unsound-skip- switch is expected to be accepted with the skip reported, and our verdict does not yet refuse one. No such proof has been run. The seams ↗sources 1 (new tab) 2 (new tab)
Decodinggreedy, at every reply positionsource (new tab)
Not checked by any callnot stated
Architecturevisible: the shape and gadget parameterssource (new tab)
Claim card in full (7 quotes)
WhatGivenClaim card
An accept meansthe reply is the committed model's own greedy reply to the prompt, on our reading of several of the publisher's sentences, none of which says it alone

Exit status 0 only when every implied link ran and chained on its own /// axis, every tag sits where its statement says it sits, every position's id /// is the prompt's or the previous position's proved argmax source (new tab)

the argmax binds the token a head segment emits to the **lowest-indexed** maximum of the logits it computed, which is the rule the model itself decodes by. source (new tab)

| # | edge | default | turned off by | source (new tab)

| 12 | each matmul's input times the pinned weights into its accumulator | bound | nothing: it is the Freivalds check itself | source (new tab)

The other preprocessed tree is the model's, and its root is not derivable by anyone but the publisher: it commits the weights, and the verifier compares it with the published manifest. source (new tab)

From our commit recordthe commitment and the files commit wrote: model-roots.json, grid.json

The model root is not derived by the verifier: it is "the publisher's commitment to weights the verifier does not hold", compared with the published model manifest (boundary-grid --expect-model-roots). source (new tab)

The verifier derives each segment's shape root from the statement's own parameters and the per-layer calibration the model manifest publishes beside the model commitment; given no calibration, the shape root is asserted by the prover and checked by nothing, and boundary-grid --verify refuses to run without one. source (new tab)

From the requestthe prompt, 512 token ids

The verifier is given the prompt's ids (--prompt), against which every prompt position's id is checked; boundary-grid --verify refuses to run without them. source (new tab)

From the proof199 files: 66 × *.claims.json, 66 × *.fixed.json, 66 × *.proof.json, plan.txt. The verifier is not given the reply and reads it from the proof. The proof's Fixed statement carries the gadget switches, so a proof made with an --unsound-skip- switch is expected to be accepted with the skip reported, and our verdict does not yet refuse one. No such proof has been run

The Fixed statement (the shape and gadget parameters, the security configuration, the tree column sizes and the pinned preprocessed root) is "produced at setup, trusted by the verifier". source (new tab)

every position of the sequence's grid: each prompt position, whose id must equal the prompt's, and each generated position, whose id must equal the token the head segment proved for the position before it source (new tab)

Decoding rulegreedy, at every reply position

the argmax binds the token a head segment emits to the lowest-indexed maximum of the logits it computed, "which is the rule the model itself decodes by" source (new tab)

Checked by no callnot stated
Architecturevisible: the shape and gadget parameters

The Fixed statement (the shape and gadget parameters, the security configuration, the tree column sizes and the pinned preprocessed root) is "produced at setup, trusted by the verifier". source (new tab)

Checks

CheckStateRecord
Runpassedrun record (new tab)
Modelpassedcontrols (new tab)
Promptpassedcontrols (new tab)
Replypassedrun record (new tab)
Verify without weightspassedverify traces (new tab)
Fidelitypassedfidelity record (new tab)
Every record and trial (16 records)
Model

What ran proved with one weight changed: 3 trials, the same input each time, and 3 refused by the verifier. An honest proof checked against the changed weights' commitment: 3 trials, the same input each time, and 3 refused by the verifier.

2026-11-12 · 3: 2 controls (new tab)
Prompt

What ran one prompt token changed: 3 trials, a new input each time, and 3 refused by the verifier.

2026-11-12 · 3: 1 control (new tab)
Reply

What ran the first reply token changed: 3 trials, the same input each time, and 3 refused by the verifier. The last reply token changed: 3 trials, the same input each time, and 3 refused by the verifier. One interior reply token changed: 3 trials, a new input each time, and 3 refused by the verifier.

2026-11-13 · 1: run record (new tab)2026-11-16 · 1: run record (new tab)2026-11-12 · 3: run record (new tab)2026-11-12 · 3: 1 control (new tab)

Cost

3 whole sessions of 5 replies · 1 × H200 · our runs only, not rerun by anyone else

tokens/s per H200, log scale
proved0.1911tokens/s per H200
served11,876tokens/s per H200
served over proved is the cost vs serving
Verify time
1,078s/replyn = 3 · 1,064 to 1,087 s/replyverifier wall time
Proof size
3,909MB/replyn = 3 · 3,902 to 3,912 MB/replyevery proof file, as written
Cost
proved3.979USD/replyserved0.00006401USD/replyH2005.345USD/GPU-hour
GPU time, hardware and serving

Serving Meta's BF16 release, vLLM 0.32.0, 1 × H200, full load at 256 concurrent requests · n = 3 timed batches, 1 run (2026-11-11 · 1 (new tab)) · the copy this row proves is under Fidelity

Proving 1 × H200, 3 whole sessions of 5 replies

Verifying boundary-grid --verify on 8 CPUs of an Intel(R) Xeon(R) Platinum 8488C, the prover's host, its GPU visible · 1,064 to 1,087 s/reply, the 3 sessions' means, n = 3

Proof size the bytes of the 199 files a proof is released as, as written: 66 × *.claims.json, 66 × *.fixed.json, 66 × *.proof.json, plan.txt

Tokens per second 0.1911tokens/s per GPU (0.1828 to 0.2031, n = 3 whole sessions, 1 × H200)

Evidence our runs only, not rerun by anyone else

Ran on Hugging Face Jobs, image sash@sha256:14defac4, harness 09a9ee78 (new tab) or 3b5197dc (new tab) or b696751a (new tab), adapter c3a508bf (new tab)

Serving GPU time
0.04311GPU-s/reply
Steady GPU time
2,348GPU-s/reply2,230 to 2,429 GPU-s/reply
Session GPU time
2,680GPU-s/reply2,521 to 2,801 GPU-s/reply
Commit GPU time
325.2GPU-s/reply288.6 to 350.0 GPU-s/reply
Proving GPU time
2,355GPU-s/reply2,232 to 2,451 GPU-s/reply
Steady cost vs serving
54,450×
Steady cost
3.485USD/reply
H200 price
5.345USD/GPU-houras of 2026-11-08

Serving ladder, 11 concurrencies

Concurrent requestsRate
10.4074replies/GPU-s
20.8082replies/GPU-s
41.573replies/GPU-s
83.012replies/GPU-s
165.567replies/GPU-s
329.456replies/GPU-s
6414.87replies/GPU-s
12820.15replies/GPU-s
256 (full load)23.20replies/GPU-s
51224.27replies/GPU-s
1,02423.12replies/GPU-s

Fidelity

top-token agreement, n = 15 prompts
SASH92.0%
3-bit copy78.1%
2-bit copy72.2%
Proved
8-bit integer weights, floating-point activations, for part of the weightsW8A8
Against the proofs
480 of 480 positions15 of 15 where the two replies partwhere a proof states the model's token, SASH's code picked it
Fidelity check
passedSASH31.00tokenspass bar1.400tokensmatched leading tokens, mean
KL to the original weights
not measured
The copies, the tests and the sources

Proved W8A8 (new tab) (8-bit weights and activations), 8 bits per weight for part of the weights, as the publisher states it (new tab). Integer weights, floating-point activations, as the publisher states: “weight matmuls are exact int8 x int8 -> int32 accumulations” (new tab)“every value flowing between ops is a bf16 value” (new tab)

RepliesKindTop-token agreementMatched leading tokensKL divergenceKL on fixed texts
8-bit copy (Q8_0)honest98.4%median 98.4 %106.1tokens0.001732nats/token0.002956nats/token
SASHthis system92.0%median 92.0 %95 % interval 89.5 to 94.5 %31.00tokensn = 15 replies · above the bar: p = 1.2 × 10−40.04372nats/tokennot measuredthe committed copy is not a file llama.cpp loads

SASH's top-token agreement and KL divergence: computed by the system's own code on the reference reply, not by the proved computation. What ran: SASH's witness extractor (harness.witness.extract --head), the program that writes the prover's input, run on the reference reply; no proof is made of these ids.

run-sash.json, teacher-forced.json (new tab)
4-bit copy (Q4_K_M)honest90.6%median 91.2 %5.267tokens0.06918nats/token0.1335nats/token
3-bit copy (Q3_K_S)damaged, the bar78.1%median 78.3 %1.400tokens0.3321nats/token0.5657nats/token
2-bit copy (Q2_K)damaged72.2%median 73.0 %1.400tokens0.5518nats/token0.9252nats/token

Original: NousResearch/Meta-Llama-3.1-8B-Instruct at d10aef7, 16-bit. It is a copy on the Hub, not Meta's own repository. Fed its own reply, the original's engine picks that reply's token at 100.0 % of positions, which is the figure of the original itself.

Engines: the original and the controls run with it in llama.cpp, one request at a time.

Prompts: 512-token windows of 8 texts from www.gutenberg.org (public domain in the United States), with no chat template.

14 tests on top-token agreement, and they set no state

the original, run again
SASH above it: p = 1.0 · below it: p = 3.1 × 10−5, the test's floor
8-bit copy (Q8_0)
SASH above it: p = 1.0 · below it: p = 3.1 × 10−5, the test's floor
4-bit copy (Q4_K_M)
SASH above it: p = 0.0058 · below it: p = 0.99
3-bit copy (Q3_K_S)
SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0
2-bit copy (Q2_K)
SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0
a quarter of blocks cut
SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0
a smaller model
SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0

Sessions

3 counted · 5 release groups in each · not counted: 2 replaced, 1 ineligible

SlotRunSessionSteady
12026-11-13 · 1 (new tab) · manifest (new tab) · log (new tab)2,801GPU-s/reply2,429GPU-s/reply
22026-11-16 · 1 (new tab) · manifest (new tab) · log (new tab)2,521GPU-s/reply2,230GPU-s/reply
32026-11-12 · 3 (new tab) · manifest (new tab) · log (new tab)2,718GPU-s/reply2,384GPU-s/reply
Sessions not counted

Replaced sessions

Why its verify ran on a host with a GPU visible and its proofs were not kept to verify again (SE-3a)

SlotRunReplaced by
12026-11-12 · 1 (new tab)2026-11-13 · 1 (new tab)
22026-11-12 · 2 (new tab)2026-11-16 · 1 (new tab)

Ineligible sessions

SlotRunWhy
22026-11-13 · 2 (new tab)the host's calibration self-test did not pass