SASH
Cost vs serving62,160×3 whole sessions of 5 replies0.1911 proved, tokens/s per H200
- Standing
- Passed
- Model
- Llama 3.1 8B InstructDense, 8.03B
- Version
- PR #36 (new tab) · not merged upstream · by this project's owner · upstream
eb95bfa: SASH's own code refused one Level 1 prompt (record (new tab))
Glyph
GradesMetMostly metPartly metNot metNot statedFailedNot decided
| IntegrityCan a fake reply pass the check? | Partly met | The limit names no verifier.Met needsa proved limit of 2−128 for boundary-grid --verify, which we runHasabout 2−100 per segment, conjectured7 missing · publisher acts, IntegrityMissingFor Mostly met
For Met
EvidencePublisher's words at 4f41667 (3)“stwo's own conjectured-soundness accounting is security_bits = pow_bits + log_blowup_factor * n_queries so blowup 2 (log_blowup_factor = 1) at 100 conjectured bits means exactly 100 FRI queries and no grinding.” (new tab)“A proof made by this prover is about one segment: one transformer layer” (new tab)“assert_eq!(cfg.security_bits(), 100, ‘security parameter drifted’);” (new tab)Our notes (1)The security parameters are constants (util.rs L14-L16), and pcs_config() asserts 100 bits (L26). The row runs the sound defaults. It does not use our reading that the limit is for the settings we run. Met needs the publisher's words there too. Why this gradePartly met: the limit is not shown to cover what we run (items 1 to 5 under Missing). It is not a theorem, and its exponent is 100. Met needs, in fullA proved limit of 2−128 on a fake reply passing the verifier we run. What it hasThe publisher states 100 conjectured bits of security per segment of a proof: a chance of about 2−100 that a fake segment passes. The figure comes from the accounting of stwo, a proof library. Conjectured means it rests on an unproven assumption, not a theorem. One segment proves one transformer layer (“A proof made by this prover is about one segment: one transformer layer” (new tab)). A reply passes through every layer of the model, but the publisher states no limit for a whole reply. The publisher names no verifier for this figure, and does not say which cheating provers it holds against. We run boundary-grid --verify, the publisher's verifier command. The publisher states that this command runs the publisher's check, verify_composed (“ Each item today
Our recordsIn each of our Model, Prompt and Reply tests, we give the verifier a proof with the wrong model, prompt or reply. Each was refused on this run. In the field's termsSoundness. Bound: 100 conjectured bits per segment (one transformer layer). Basis: conjectured, stwo's accounting. Stated for: no verifier named. We run: boundary-grid --verify (no bound claimed for it). Adversary: not said. Unit: segment. Batch: not said. Ours is 1 reply. Parameters: row (our reading). Conditions shown for our files: no. |
|---|---|---|
| FidelityDoes the proved model answer like the original? | Partly met | Our test does not show an average above 55.7 tokens, which Mostly met needs.Met needsan average shown above 309.0 tokens, one request at a timeHas31.00 matched tokens (8-bit copy 106.1, 4-bit copy 5.267)2 missing · publisher acts, FidelityMissingFor Mostly met
For Met
Why this gradeMet needs, in fullAn average above 309.0 matched tokens, shown by our test, in a run of one request at a time. What it hasWe sent 15 prompts to SASH and to the original model (Llama 3.1 8B Instruct, 16-bit). On average, SASH's replies match the original's for their first 31.00 tokens, up to the first difference (512 at most). For scale, we scored other runs on the same prompts: the original model run a second time scores 512.0 (every token, since 512 is the most a reply has), our 8-bit copy 106.1, our 4-bit copy 5.267, and a copy we damaged on purpose (3-bit) 1.400, the pass bar every system must beat. The publisher states that SASH proves a copy with 8-bit integer weights for part of its weights, and states no width for the rest. Values between its layers are floating-point numbers. SASH's copy and our 8-bit copy (Q8_0, a llama.cpp format) are different 8-bit copies. Our test measures the gap, not its cause. Our run sent one request at a time, as Met needs. Each item today
Our recordsOur fidelity run on Llama 3.1 8B Instruct, 15 prompts, each reply scored against the original's. In the field's termsQuantization error, in matched leading tokens (mean, n = 15). Above T_upper, 55.7 (Mostly met): not shown. Above T_green, 309.0 (Met): not shown. Test: paired sign-flip per prompt (exact), alpha 0.05. Proved: 8-bit integer weights, floating-point activations, for part of the weights. Decoding: exact match (only the model's top token). Requests at a time (slots): 1 for the whole run (llama.cpp). |
| Formally verifiedHas a computer checked that fake replies are refused? | Not stated | We found no computer-checked theorem named in the 4 documents we read.Met needsa rebuilt theorem reaching boundary-grid --verify, which we runHasnothing stated4 missing · publisher 3, us 1, Formally verifiedMissingFor Partly met
For Mostly met
For Met
Why this gradeMet needs, in fullA computer-checked theorem that the verifier we run refuses fake replies, rebuilt by us. What it hasThe 4 documents we read at the version we ran (4f41667) name no computer-checked theorem. Our recordsOur Model, Prompt and Reply tests passed on this run. In the field's termsMachine-checked soundness proof. Tool, theorem and trusted base: not stated. Scope: the scope table further down this page ↓ |
| Trust assumptionsDoes checking trust a setup or a party? | Mostly metour reading | No single sentence of the publisher's says it.Met needsthe publisher's words: cryptography only, no setup secretHascryptography only, no setup secret, on our reading1 missing · publisher acts, Trust assumptionsMissingFor Met
EvidenceFrom the publisher's words and our reading Publisher's words at 4f41667 (6)“stwo's own conjectured-soundness accounting is security_bits = pow_bits + log_blowup_factor * n_queries so blowup 2 (log_blowup_factor = 1) at 100 conjectured bits means exactly 100 FRI queries and no grinding.” (new tab)“The preprocessed trace is two committed trees (a shape tree whose root the verifier derives from the public parameters, and a model tree whose root is compared with the published model commitment).” (new tab)“They are committed as a tree of their own, and what says they are the columns the public parameters give is that the verifier builds that tree itself and compares its root with the root the statement declares.” (new tab)“security (the FRI parameters and the hash)” (new tab)“Fixed — produced at setup, trusted by the verifier: the shape and gadget parameters that determine every column size and selector, the security config, the tree column sizes, and the pinned preprocessed root (every table, selector, weight and scale column lives in tree 0, and B4 §3 / B5 / B6 all showed a corrupted table is invisible to everything else).” (new tab)“The verifier (verify_composed) takes those two byte strings and the serialized proof, and nothing else” (new tab)Our notes (1)We apply the test FlashProve's row meets: we read a petal as transparent when a sentence among its quotes says what soundness rests on and none of it is a setup that produced a secret (GL-10, amended 2026-11-15). SASH states its soundness as stwo's conjectured accounting over the FRI parameters (util.rs L9-L12). It says the setup's two trees are a shape tree the verifier derives from the public parameters and a model tree compared with the published model commitment (prover/README.md L117-L119, docs/statement.md L127-L129). The Fixed statement it calls trusted by the verifier is those parameters and roots (compose.rs L7-L11). No sentence says in words that no setup produced a secret, so we read the petal as transparent and grade it Mostly met at most, as FlashProve's is. No test of ours bears on it Why this gradeMet needs, in fullThe publisher's own sentence that checking needs no secret setup and no trusted party. What it hasThe publisher states that the verifier takes two byte strings and the proof, and nothing else. We read this sentence to mean that anyone can check a proof, but the publisher does not say so in words. (the publisher's words: “The verifier ( The publisher states its soundness as stwo's conjectured estimate (see Integrity). Its setup commits two trees: a shape tree, whose root the verifier builds itself from the public settings, and a model tree, whose root the verifier compares with the published commitment. It calls some values So, on our reading, soundness rests on cryptographic assumptions only, and no setup step produced a secret. But the publisher does not say this in one sentence. Each item today
In the field's termsTrusted setup or party. Rests on: transparent (no setup made a secret) (our reading). Publicly checkable: yes. |
Completeness: 15 of 15 replies accepted, our Run check
Stated security, scope card
What is verified, and for which code, as the publisher states it. It sets no state.
| What | Bound, as stated | Protocol | The verifier we run | The prover we run |
|---|---|---|---|---|
| Soundness | 100 conjectured bits (new tab) | not stated | not stated | does not apply |
| Zero knowledge | not stated | not stated | does not apply | not stated |
| Completeness | not stated | not stated | not stated | not stated |
Statement
| Verifier | boundary-grid --verify | claim card (new tab) |
|---|---|---|
| An accept means | the committed model's own greedy reply to the prompt (our reading: no single sentence states it) | sources 1 (new tab) 2 (new tab) 3 (new tab) 4 (new tab) 5 (new tab) |
| From our commit record | the commitment, 2 files | sources 1 (new tab) 2 (new tab) |
| From the request | the prompt (512 token ids) | source (new tab) |
| From the proof | 199 files, and the verifier reads the reply from themThe proof's Fixed statement carries the gadget switches, so a proof made with an --unsound-skip- switch is expected to be accepted with the skip reported, and our verdict does not yet refuse one. No such proof has been run. The seams ↗ | sources 1 (new tab) 2 (new tab) |
| Decoding | greedy, at every reply position | source (new tab) |
| Not checked by any call | not stated | |
| Architecture | visible: the shape and gadget parameters | source (new tab) |
Claim card in full (7 quotes)
| What | Given | Claim card |
|---|---|---|
| An accept means | the reply is the committed model's own greedy reply to the prompt, on our reading of several of the publisher's sentences, none of which says it alone | Exit status 0 only when every implied link ran and chained on its own /// axis, every tag sits where its statement says it sits, every position's id /// is the prompt's or the previous position's proved argmax source (new tab) the argmax binds the token a head segment emits to the **lowest-indexed** maximum of the logits it computed, which is the rule the model itself decodes by. source (new tab) | # | edge | default | turned off by | source (new tab) | 12 | each matmul's input times the pinned weights into its accumulator | bound | nothing: it is the Freivalds check itself | source (new tab) The other preprocessed tree is the model's, and its root is not derivable by anyone but the publisher: it commits the weights, and the verifier compares it with the published manifest. source (new tab) |
| From our commit record | the commitment and the files commit wrote: model-roots.json, grid.json | The model root is not derived by the verifier: it is "the publisher's commitment to weights the verifier does not hold", compared with the published model manifest (boundary-grid --expect-model-roots). source (new tab) The verifier derives each segment's shape root from the statement's own parameters and the per-layer calibration the model manifest publishes beside the model commitment; given no calibration, the shape root is asserted by the prover and checked by nothing, and boundary-grid --verify refuses to run without one. source (new tab) |
| From the request | the prompt, 512 token ids | The verifier is given the prompt's ids (--prompt), against which every prompt position's id is checked; boundary-grid --verify refuses to run without them. source (new tab) |
| From the proof | 199 files: 66 × *.claims.json, 66 × *.fixed.json, 66 × *.proof.json, plan.txt. The verifier is not given the reply and reads it from the proof. The proof's Fixed statement carries the gadget switches, so a proof made with an --unsound-skip- switch is expected to be accepted with the skip reported, and our verdict does not yet refuse one. No such proof has been run | The Fixed statement (the shape and gadget parameters, the security configuration, the tree column sizes and the pinned preprocessed root) is "produced at setup, trusted by the verifier". source (new tab) every position of the sequence's grid: each prompt position, whose id must equal the prompt's, and each generated position, whose id must equal the token the head segment proved for the position before it source (new tab) |
| Decoding rule | greedy, at every reply position | the argmax binds the token a head segment emits to the lowest-indexed maximum of the logits it computed, "which is the rule the model itself decodes by" source (new tab) |
| Checked by no call | not stated | |
| Architecture | visible: the shape and gadget parameters | The Fixed statement (the shape and gadget parameters, the security configuration, the tree column sizes and the pinned preprocessed root) is "produced at setup, trusted by the verifier". source (new tab) |
Checks
| Check | State | Record |
|---|---|---|
| Run | passed | run record (new tab) |
| Model | passed | controls (new tab) |
| Prompt | passed | controls (new tab) |
| Reply | passed | run record (new tab) |
| Verify without weights | passed | verify traces (new tab) |
| Fidelity | passed | fidelity record (new tab) |
Every record and trial (16 records)
- Run
- 2026-11-13 · 1: run record (new tab)2026-11-16 · 1: run record (new tab)2026-11-12 · 3: run record (new tab)
- Model
What ran proved with one weight changed: 3 trials, the same input each time, and 3 refused by the verifier. An honest proof checked against the changed weights' commitment: 3 trials, the same input each time, and 3 refused by the verifier.
2026-11-12 · 3: 2 controls (new tab)- Prompt
What ran one prompt token changed: 3 trials, a new input each time, and 3 refused by the verifier.
2026-11-12 · 3: 1 control (new tab)- Reply
What ran the first reply token changed: 3 trials, the same input each time, and 3 refused by the verifier. The last reply token changed: 3 trials, the same input each time, and 3 refused by the verifier. One interior reply token changed: 3 trials, a new input each time, and 3 refused by the verifier.
2026-11-13 · 1: run record (new tab)2026-11-16 · 1: run record (new tab)2026-11-12 · 3: run record (new tab)2026-11-12 · 3: 1 control (new tab)- Verify without weights
- 2026-11-13 · 1: verify traces (new tab)2026-11-13 · 1: re-verify record (new tab)2026-11-16 · 1: verify traces (new tab)2026-11-16 · 1: re-verify record (new tab)2026-11-12 · 3: verify traces (new tab)2026-11-12 · 3: re-verify record (new tab)
- Fidelity
- fidelity record (new tab)
Cost
3 whole sessions of 5 replies · 1 × H200 · our runs only, not rerun by anyone else
- Verify time
- 1,078s/replyn = 3 · 1,064 to 1,087 s/replyverifier wall time
- Proof size
- 3,909MB/replyn = 3 · 3,902 to 3,912 MB/replyevery proof file, as written
- Cost
- proved3.979USD/replyserved0.00006401USD/replyH2005.345USD/GPU-hour
GPU time, hardware and serving
Serving Meta's BF16 release, vLLM 0.32.0, 1 × H200, full load at 256 concurrent requests · n = 3 timed batches, 1 run (2026-11-11 · 1 (new tab)) · the copy this row proves is under Fidelity
Proving 1 × H200, 3 whole sessions of 5 replies
Verifying boundary-grid --verify on 8 CPUs of an Intel(R) Xeon(R) Platinum 8488C, the prover's host, its GPU visible · 1,064 to 1,087 s/reply, the 3 sessions' means, n = 3
Proof size the bytes of the 199 files a proof is released as, as written: 66 × *.claims.json, 66 × *.fixed.json, 66 × *.proof.json, plan.txt
Tokens per second 0.1911tokens/s per GPU (0.1828 to 0.2031, n = 3 whole sessions, 1 × H200)
Evidence our runs only, not rerun by anyone else
Ran on Hugging Face Jobs, image sash@sha256:14defac4, harness 09a9ee78 (new tab) or 3b5197dc (new tab) or b696751a (new tab), adapter c3a508bf (new tab)
- Serving GPU time
- 0.04311GPU-s/reply
- Steady GPU time
- 2,348GPU-s/reply2,230 to 2,429 GPU-s/reply
- Session GPU time
- 2,680GPU-s/reply2,521 to 2,801 GPU-s/reply
- Commit GPU time
- 325.2GPU-s/reply288.6 to 350.0 GPU-s/reply
- Proving GPU time
- 2,355GPU-s/reply2,232 to 2,451 GPU-s/reply
- Steady cost vs serving
- 54,450×
- Steady cost
- 3.485USD/reply
- H200 price
- 5.345USD/GPU-houras of 2026-11-08
Serving ladder, 11 concurrencies
| Concurrent requests | Rate |
|---|---|
| 1 | 0.4074replies/GPU-s |
| 2 | 0.8082replies/GPU-s |
| 4 | 1.573replies/GPU-s |
| 8 | 3.012replies/GPU-s |
| 16 | 5.567replies/GPU-s |
| 32 | 9.456replies/GPU-s |
| 64 | 14.87replies/GPU-s |
| 128 | 20.15replies/GPU-s |
| 256 (full load) | 23.20replies/GPU-s |
| 512 | 24.27replies/GPU-s |
| 1,024 | 23.12replies/GPU-s |
Fidelity
- Proved
- 8-bit integer weights, floating-point activations, for part of the weightsW8A8
- Against the proofs
- 480 of 480 positions15 of 15 where the two replies partwhere a proof states the model's token, SASH's code picked it
- Fidelity check
- passedSASH31.00tokenspass bar1.400tokensmatched leading tokens, mean
- KL to the original weights
- not measured
The copies, the tests and the sources
Proved W8A8 (new tab) (8-bit weights and activations), 8 bits per weight for part of the weights, as the publisher states it (new tab). Integer weights, floating-point activations, as the publisher states: “weight matmuls are exact int8 x int8 -> int32 accumulations” (new tab)“every value flowing between ops is a bf16 value” (new tab)
| Replies | Kind | Top-token agreement | Matched leading tokens | KL divergence | KL on fixed texts | |
|---|---|---|---|---|---|---|
| 8-bit copy (Q8_0) | honest | 98.4%median 98.4 % | 106.1tokens | 0.001732nats/token | 0.002956nats/token | |
| SASH | this system | 92.0%median 92.0 %95 % interval 89.5 to 94.5 % | 31.00tokensn = 15 replies · above the bar: p = 1.2 × 10−4 | 0.04372nats/token | not measuredthe committed copy is not a file llama.cpp loads | |
SASH's top-token agreement and KL divergence: computed by the system's own code on the reference reply, not by the proved computation. What ran: SASH's witness extractor (harness.witness.extract --head), the program that writes the prover's input, run on the reference reply; no proof is made of these ids. run-sash.json, teacher-forced.json (new tab) | ||||||
| 4-bit copy (Q4_K_M) | honest | 90.6%median 91.2 % | 5.267tokens | 0.06918nats/token | 0.1335nats/token | |
| 3-bit copy (Q3_K_S) | damaged, the bar | 78.1%median 78.3 % | 1.400tokens | 0.3321nats/token | 0.5657nats/token | |
| 2-bit copy (Q2_K) | damaged | 72.2%median 73.0 % | 1.400tokens | 0.5518nats/token | 0.9252nats/token | |
Original: NousResearch/Meta-Llama-3.1-8B-Instruct at d10aef7, 16-bit. It is a copy on the Hub, not Meta's own repository. Fed its own reply, the original's engine picks that reply's token at 100.0 % of positions, which is the figure of the original itself.
Engines: the original and the controls run with it in llama.cpp, one request at a time.
Prompts: 512-token windows of 8 texts from www.gutenberg.org (public domain in the United States), with no chat template.
14 tests on top-token agreement, and they set no state
- the original, run again
- SASH above it: p = 1.0 · below it: p = 3.1 × 10−5, the test's floor
- 8-bit copy (Q8_0)
- SASH above it: p = 1.0 · below it: p = 3.1 × 10−5, the test's floor
- 4-bit copy (Q4_K_M)
- SASH above it: p = 0.0058 · below it: p = 0.99
- 3-bit copy (Q3_K_S)
- SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0
- 2-bit copy (Q2_K)
- SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0
- a quarter of blocks cut
- SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0
- a smaller model
- SASH above it: p = 3.1 × 10−5, the test's floor · below it: p = 1.0
Sessions
3 counted · 5 release groups in each · not counted: 2 replaced, 1 ineligible
| Slot | Run | Session | Steady |
|---|---|---|---|
| 1 | 2026-11-13 · 1 (new tab) · manifest (new tab) · log (new tab) | 2,801GPU-s/reply | 2,429GPU-s/reply |
| 2 | 2026-11-16 · 1 (new tab) · manifest (new tab) · log (new tab) | 2,521GPU-s/reply | 2,230GPU-s/reply |
| 3 | 2026-11-12 · 3 (new tab) · manifest (new tab) · log (new tab) | 2,718GPU-s/reply | 2,384GPU-s/reply |
Sessions not counted
Replaced sessions
Why its verify ran on a host with a GPU visible and its proofs were not kept to verify again (SE-3a)
| Slot | Run | Replaced by |
|---|---|---|
| 1 | 2026-11-12 · 1 (new tab) | 2026-11-13 · 1 (new tab) |
| 2 | 2026-11-12 · 2 (new tab) | 2026-11-16 · 1 (new tab) |
Ineligible sessions
| Slot | Run | Why |
|---|---|---|
| 2 | 2026-11-13 · 2 (new tab) | the host's calibration self-test did not pass |