FlashProve
Cost vs serving1,317×3 whole sessions of 120 replies2.027 proved, tokens/s per H200
- Standing
- Passed
- Model
- Llama 4 Maverick 17B-128E InstructMixture of experts, 400B total, 17B active
Glyph
GradesMetMostly metPartly metNot metNot statedFailedNot decided
| IntegrityCan a fake reply pass the check? | Partly metour reading | The limit is proved for the Lean verifier, not cp_verify.Met needsa proved limit of 2−128 for cp_verify, which we runHasq/2124 + 2−101 per proof, proved5 missing · publisher acts, IntegrityMissingFor Mostly met
For Met
EvidenceFrom the publisher's words and our reading Publisher's words at 9e2bdd8 (15)“The Lean verifier accepts a proof of a false claim with probability at most about q/2^124 + 2^-101 after q hash queries (about 124 bits per query; not ‘128 bits’)” (new tab)“Lean's kernel has checked the proof.” (new tab)“The bound is for one commitment, one prompt, one output, one proof and one call ofverify.” (new tab)“One proof. N proofs handed in at once under one commitment, or N members of a batch, cost N times that error, taken at the whole list's hash queries; for proofs checked one after another, or members checked in separate sessions, no bound is stated.” (new tab)“for separate sessions no theorem states it.” (new tab)“against a committer and a prover who may both cheat” (new tab)“so nothing is claimed against a quantum adversary” (new tab)“One proof format and model size: full size, not the small examples” (new tab)“Neither figure, 124 bits per query or 128 bits per round, is claimed for the C++ programs.” (new tab)“At all other files in scope nothing is known, so there the theorem may hold only because verify rejects everything.” (new tab)“The proofs from the runs on real weights were checked by cp_verify alone; none has been put to the Lean verifier” (new tab)“public files that are fixed before the hash function is sampled (a provider that tunes them to the real hash function is outside the statement)” (new tab)“No file shows that they were fixed before the provider could tune them to the hash function: that you grant.” (new tab)“The C++ verifier and the CUDA prover are written to follow the same protocol and are tested against it, not themselves proved.” (new tab)“The C++ programs' printed ‘bits’ are their own accounting, not a theorem” (new tab)Our notes (2)Classical adversaries only (THEOREM.md L394-L395), and the files-fixed-before-the-hash condition is counted under the conditions clause. The row runs the full-size format at exact match. Whether its own files are in scope falls under the conditions clause, and the bound is stated for the Lean verifier and not claimed for cp_verify, which the row runs (the program clause). The grade uses our reading that the limit holds against any cheating prover. It does not use our reading that the limit is for the settings we run. Met needs the publisher's words there too. Why this gradePartly met: the limit is not shown to cover what we run (items 1 to 3 under Missing). Its exponents, as written, are 124 and 101. Met needs, in fullA proved limit of 2−128 on a fake reply passing the verifier we run. What it hasThe publisher proves, and states that Lean has checked, a limit on the chance that a fake reply passes: at most q/2124 + 2−101 per proof. Here q counts the hash queries a cheater makes, each a run of the hash function while it searches for a proof that passes. The other part is fixed, but the first grows with q. It reaches 2−100 at 224 queries and 2−80 at 244 queries. Each reply has its own proof, and we make replies in batches of 24 (item 2 below). The theorem is about the Lean verifier, a verifier the publisher wrote in Lean and proves things about. We run cp_verify, the publisher's C++ verifier (“Anyone can check the proof on an ordinary CPU in seconds with Our reading, on one point the grade uses (no single sentence of the publisher's says it): The publisher states the limit “against a committer and a prover who may both cheat” (the committer publishes the commitment, and the prover makes the proofs). Our rule asks for a limit against any cheating prover, one that runs its own machine as it likes, and the limit must not rely on trusted hardware or an honest outside party. We read the publisher's words as that limit. The publisher adds that nothing is claimed against a quantum computer, but our rule does not ask about one. Each item today
Our recordsIn each of our Model, Prompt and Reply tests, we give the verifier a proof with the wrong model, prompt or reply. Each was refused on this run. In the field's termsSoundness. Bound: at most q/2124 + 2−101 per proof, after q hash queries. The first term is 2−100 at q = 224 and 2−80 at q = 244. Basis: proved. Stated for: the Lean verifier. We run: cp_verify (no bound claimed for it). Adversary: any prover (our reading). Unit: proof, one reply per proof. Batch: together. Ours is 24 replies, each verified in its own call. Parameters: row (our reading). Conditions shown for our files: no. |
|---|---|---|
| FidelityDoes the proved model answer like the original? | Partly met | Our test does not show an average above 48.0 tokens, which Mostly met needs.Met needsan average shown above 283.0 tokens, one request at a timeHas29.66 matched tokens (8-bit copy 65.71, 4-bit copy 30.29)3 missing · publisher 2, us 1, FidelityMissingFor Mostly met
For Met
Why this gradeMet needs, in fullAn average above 283.0 matched tokens, shown by our test, in a run of one request at a time. What it hasWe sent 360 prompts to FlashProve and to the original model (Llama 4 Maverick 17B-128E Instruct, 16-bit). On average, FlashProve's replies match the original's for their first 29.66 tokens, up to the first difference (512 at most). For scale, we scored other runs on the same prompts: the original model run a second time scores 500.3, our 8-bit copy 65.71, our 4-bit copy 30.29, and a copy we damaged on purpose (3-bit) 11.15, the pass bar every system must beat. FlashProve proves a copy with integer weights of 4.84 bits each on average, counting the weights each token uses (Llama 4 Maverick uses about 17B of its 400B weights for each token). Values between its layers are fixed-point numbers (integers with a fixed scale). Each item today
Our recordsOur fidelity run on Llama 4 Maverick 17B-128E Instruct, 360 prompts, each reply scored against the original's. In the field's termsQuantization error, in matched leading tokens (mean, n = 360). Above T_upper, 48.0 (Mostly met): not shown. Above T_green, 283.0 (Met): not shown. Test: paired sign-flip per prompt, alpha 0.05. Proved: 4.84-bit integer weights, fixed-point activations. Decoding: exact match (only the model's top token). Requests at a time (slots): no count for the original and its rerun (vLLM, one call) and 16 for the copies (llama.cpp). |
| Formally verifiedHas a computer checked that fake replies are refused? | Mostly metour reading | The theorem stops at the Lean verifier, short of cp_verify.Met needsa rebuilt theorem reaching cp_verify, which we runHasa Lean theorem of the method, rebuilt by us3 missing · publisher 2, us 1, Formally verifiedMissingFor Met
EvidenceFrom the publisher's words, our rebuild, our reading (2 points) Publisher's words at 9e2bdd8 (13)“Soundness is proved, in Lean, for the verifier written in Lean” (new tab)“At each position it is the greedy next token after the prompt and the output's own earlier tokens, as an integer program computes it, not the floating-point model.” (new tab)“That this interactive oracle proof is the protocol of the C++ programs is not proved” (new tab)“Scale files in scope can switch off any of the model's blocks, up to all of them (worked out by hand, not proved).” (new tab)“No theorem describes the five constant look-up tables in the Lean text (cosine, sine, the exponential, SiLU and the sigmoid).” (new tab)“the kernel checks proofs, not what a definition means” (new tab)“‘The model’ means the extracted weights, which no theorem says are the onescp_commit put in” (new tab)“the system in the statement of the soundness theorem on verify has a constant committer (a committer that returns one constant whatever the weights, and so commits to nothing; the committer of the six lines is another: the one that compile makes (lean/Johnson/Face/Compiler.lean), which encodes the weights and hashes the key) where its honest committer goes, so of that system itself none of it can be stated with any content” (new tab)“The protocol's soundness is proved in Lean 4.” (new tab)“the pinned version, 4.29.1, is older than fixes of publicly known soundness bugs in it” (new tab)“leanprover/lean4:v4.29.1” (new tab)“Lean's kernel and three standard axioms: propext, Classical.choice, Quot.sound.” (new tab)“Five of the files that this function takes come from a parser outside Lean, of which nothing is proved” (new tab)Our notes (2)The theorem covers the Lean protocol and states the model as an integer program (THEOREM.md L231-233). The C++ verifier the row runs is tested, not proved (what-is-proved.md L7-8), so the reach stops short of the verifier code this row runs. The model statement is the publisher's Lean integer model, and its tables and its reach at a given scale file are not proved (THEOREM.md L229-L231, L242-L243). Read 2026-11-15: the conclusion is about the publisher's integer model, a reference definition of the model's computation, at the weights the extractor reads. No theorem ties those weights to the ones cp_commit committed (THEOREM.md L227-L228), and at files other than the stored examples the theorem may hold only because verify rejects everything (L314-L315, under Integrity's conditions clause). Also, the system in the soundness theorem's statement has a constant committer, which commits to nothing, so of that system none of it can be stated with any content (L700-L704). So the conclusion is about a reference definition of the model, as a step above Partly met needs, but it is not tied to the weights this row commits. Why this gradeMostly met, on our reading of the reach and the conclusion. It does not reach cp_verify. Met needs, in fullA computer-checked theorem that the verifier we run refuses fake replies, rebuilt by us. What it hasThe publisher proves, in Lean, a theorem that the Lean verifier refuses fake replies, up to the chance limit under Integrity. Lean is a program that checks every step of a proof. The theorem's statement includes the model's computation (our reading, below). We rebuilt it at the version we ran (9e2bdd8) with the Lean version the publisher pins, 4.29.1. It checked: the publisher's check command exited with code 0 after 68,494 s (about 19 hours). It uses only Lean's three standard axioms, the basic assumptions any Lean proof may use, so no step is skipped. Lean itself is trusted, and the limits the publisher states are listed below. It does not reach cp_verify, the publisher's C++ verifier, which we run. The publisher states that cp_verify is tested, not proved. Input-reading code sits outside it: “Five of the files that this function takes come from a parser outside Lean, of which nothing is proved” (new tab), as the publisher states. We hold no record of a review of the theorem's statement by an outside Lean expert. Our reading, on two points the grade uses (no single sentence of the publisher's says it): (1) The theorem covers the whole method, the model's computation included. (2) Its conclusion is about the model: about the publisher's integer model (a written-out definition, in Lean, of what the model computes) at the weights read from the files. No theorem ties those weights to the ones cp_commit committed (the first limit below). But our rules accept a conclusion about such a definition as one about the model, and they do not ask for a theorem that ties its weights to the committed ones. Each item today
Our recordsOur rebuild: Outside the grade: we ran the Lean verifier on 5 honest proofs from our runs, at the files we run (2 from counted sessions). It accepted 5 of 5. As a check that it can refuse, we also gave it one proof with a commitment that no run of ours made, and it refused that proof. The trial script's wall clock reads 1,237 to 1,530 s per proof (n = 5). Record ↗ (new tab) Our Model, Prompt and Reply tests passed on this run. Limits the publisher states, which the grade does not count
In the field's termsMachine-checked soundness proof. Reach: the protocol, model included (our reading). Theorem: |
| Trust assumptionsDoes checking trust a setup or a party? | Mostly metour reading | No single sentence of the publisher's says it.Met needsthe publisher's words: cryptography only, no setup secretHascryptography only, no setup secret, on our reading1 missing · publisher acts, Trust assumptionsMissingFor Met
EvidenceFrom the publisher's words and our reading Publisher's words at 9e2bdd8 (7)“Six theorems are proved in Lean of one committer, prover and verifier,compile (flashProve setup), at every public setup setup (PublicSetup: three public files, what the reader returns at them, and FilesInUse).” (new tab)“Sound says that there is one extractor such that, whatever the committer and the prover, verify accepts a false claim with probability at most the bound at q = a + b.” (new tab)“public files that are fixed before the hash function is sampled (a provider that tunes them to the real hash function is outside the statement)” (new tab)“the soundness theorem on verify has no hypothesis beyond those in its statement, so nothing more is assumed by that road.” (new tab)“The random-oracle model for the hash, with a second, fixed hash function, built on BLAKE2b's compression function, assumed independent of it, against classical adversaries.” (new tab)“the random oracle and the second hash function belong to the statement, not to the proof's axioms.” (new tab)“Anyone can check the proof on an ordinary CPU in seconds with cp_verify, without the weights.” (new tab)Our notes (1)The publisher states soundness at every public setup of three public files, against any committer and prover, with the random-oracle model as its assumption. No sentence says in words that no setup produced a secret, so we read the petal as transparent and grade it Mostly met at most (GL-10, amended 2026-11-15). No test of ours bears on it Why this gradeMet needs, in fullThe publisher's own sentence that checking needs no secret setup and no trusted party. What it hasAnyone can check a proof, as the publisher states (“Anyone can check the proof on an ordinary CPU in seconds with The publisher proves its theorems at every public setup (three public files it names), against any committer and prover. It assumes the random-oracle model, which treats the hash function as perfectly random. It also assumes that a second, fixed hash function, built on BLAKE2b's compression function, is independent of the first. The publisher's theorems are stated against classical adversaries, not quantum ones. These are idealizations of hash functions, not proved properties of them. On our reading, none of them needs a secret or a trusted party, and we count them among cryptographic assumptions. The theorem's condition that the public files are fixed before the hash function is drawn is counted under Integrity. So, on our reading, soundness rests on cryptographic assumptions only, and no setup step produced a secret. But the publisher does not say this in one sentence. Each item today
In the field's termsTrusted setup or party. Rests on: transparent (no setup made a secret) (our reading). Publicly checkable: yes. |
Completeness: 360 of 360 replies accepted, our Run check
Stated security, scope card
- Stated security
- For the Lean verifier: a false claim is accepted with probability at most about q/2^124 + 2^-101 after q hash queries. Not claimed for cp_verify, which this row runs: “The C++ programs' printed bits are their own accounting, not a theorem”. (new tab) Its notion, as the publisher names it: “
Soundis index-binding knowledge soundness, in its witness-free case, as defined inlean/ZKCP/Defs.lean(HasIndexBinding; the terms: TERMS.md), for a claim (a prompt with an output that is said to be the model's)” (new tab)“This is binding for the language, not knowledge soundness: the theorem does not show that the prover knows the weights or ran the model.” (new tab)
What is verified, and for which code, as the publisher states it. It sets no state.
| What | Bound, as stated | Protocol | The verifier we run | The prover we run |
|---|---|---|---|---|
| Soundness | q/2^124 + 2^-101 (new tab) | machine-checked (new tab) | tested by the publisher, not proved (new tab) | does not apply |
| Zero knowledge | at or below 2^-154 within 2^64 hash queries and 688,127 replies (new tab) | machine-checked (new tab) | does not apply | tested by the publisher, not proved (new tab) |
| Completeness | rejected with probability at most 2^-40 (new tab) | machine-checked (new tab) | tested by the publisher, not proved (new tab) | tested by the publisher, not proved (new tab) |
Statement
| Verifier | cp_verify | claim card (new tab) |
|---|---|---|
| An accept means | the committed model's own greedy reply to the prompt (stated) | sources 1 (new tab) 2 (new tab) |
| From our commit record | the commitment, 5 files | sources 1 (new tab) 2 (new tab) |
| From the request | the prompt (512 token ids), our 512-token reply | source (new tab) |
| From the proof | 1 file | source (new tab) |
| Decoding | greedy, at every reply position | source (new tab) |
| Not checked by any call | listed by the publisher | source (new tab) |
| Architecture | visible: the architecture and every size | source (new tab) |
Claim card in full (7 quotes)
Erratum: the card's relation and coverage.tokens_covered describe the v0.1 range: r = T - 1, e = T, one checked token after a 1,024-token window. At Level 1: “For a prompt of P tokens and a reply of R, r = P − 1 and e = P + R − 1.” (new tab)
| What | Given | Claim card |
|---|---|---|
| An accept means | the reply is the committed model's own greedy reply to the prompt, as the publisher states | Under exact match, a `YES` [is meant to show](../docs/limits.md#what-a-yes-covers) that each checked token is the committed integer model's top choice at its position source (new tab) For a prompt of P tokens and a reply of R, r = P − 1 and e = P + R − 1. source (new tab) |
| From our commit record | the commitment and the files commit wrote: policy.txt, KL.bin, layout.txt, key.bin, 0.bin | The commitment ID comes from "a party your user trusts, never from the server being checked or from files sent with the proof". source (new tab) "The commitment ID is a hash, in 64 hex digits, of the configuration, the Merkle root and shape of the first weight table, a masked encoding of the weights, and the out-of-domain (OOD) seeds and answers. [...] the ID covers them by design, but the soundness theorem on `verify` is stated at given public files and does not itself say that one ID cannot verify under two sets of them. The ID does not cover the leaf files or the key file's proofs, which `cp_verify` checks on reading." source (new tab) |
| From the request | the prompt, 512 token ids, and the 512-token reply from our record | "Only the named output positions are covered, and a reply stopped short verifies as a shorter one. [...] A proof names no client or time." source (new tab) |
| From the proof | 1 file: response.proof | "Some bytes of an honest proof, or of a commitment's files, can be altered and the proof still accepted, for the same true claim; non-malleability is not claimed, and an accept does not show that this exact file came from the prover." source (new tab) |
| Decoding rule | greedy, at every reply position | "the policy under which every checked token must be the integer model's top choice. It is `cp_verify`'s default." The published commitment's acceptance policy is exact match. source (new tab) |
| Checked by no call | "The client must trust what no call checks: the commitment ID, the expected policy and its numbers, the tokenizer and chat template, and `cp_verify` itself." source (new tab) | |
| Architecture | visible: the architecture and every size | The declared list, "At least eleven things are visible": each reply, whether a request was served or failed, time, the architecture and every size, settings fitted to the weights, that `cp_commit`'s checks passed, that a batch failed or the service stopped, how much a provider has served and when, who shared a batch and how many, lengths, places in the batch and nonces, masked sums of the weights. "The list says what is visible by design, not that everything else is hidden." In Lean's theorem, one bit per request (served or refused) is revealed, "and it depends on the weights". source (new tab) |
Checks
| Check | State | Record |
|---|---|---|
| Run | passed | run record (new tab) |
| Model | passed | controls (new tab) |
| Prompt | passed | controls (new tab) |
| Reply | passed | run record (new tab) |
| Verify without weights | passed | verify traces (new tab) |
| Fidelity | passed | fidelity record (new tab) |
Every record and trial (14 records)
- Run
- 2026-11-11 · 1: run record (new tab)2026-11-12 · 1: run record (new tab)2026-11-12 · 2: run record (new tab)
- Model
What ran proved with one weight changed: 3 trials, the same input each time, and in 3, the prover made no proof, as its publisher documents. An honest proof checked against the changed weights' commitment: 3 trials, the same input each time, and 3 refused by the verifier.
Outside the counted suite the verifier accepted 3 of 3 honest proofs (2 from counted sessions). Checked against another session's commitment of the same weights: 3 of 3 refused. So a refusal against the changed weights' commitment does not by itself show that the verifier tells weights apart, because it also refuses a second commitment of the same weights. trial record (new tab)
2026-11-11 · 1: 2 controls (new tab)2026-11-13 · 1: 1 control (new tab)- Prompt
What ran one prompt token changed: 3 trials, a new input each time, and 3 refused by the verifier.
2026-11-11 · 1: 1 control (new tab)- Reply
What ran the first reply token changed: 3 trials, the same input each time, and 3 refused by the verifier. The last reply token changed: 3 trials, the same input each time, and 3 refused by the verifier. One interior reply token changed: 3 trials, a new input each time, and 3 refused by the verifier.
Outside the counted suite the verifier accepted 1 of 1 honest proof. One reply token changed, at every reply position in turn: 512 of 512 refused. trial record (new tab)
2026-11-11 · 1: run record (new tab)2026-11-11 · 1: 1 control (new tab)2026-11-12 · 1: run record (new tab)2026-11-12 · 2: run record (new tab)- Verify without weights
- 2026-11-11 · 1: verify traces (new tab)2026-11-12 · 1: verify traces (new tab)2026-11-12 · 2: verify traces (new tab)
- Fidelity
- fidelity record (new tab)
Cost
3 whole sessions of 120 replies · 1 × H200 · our runs only, not rerun by anyone else
- Cost vs serving, in two parts
- commit and proving118.2×generation1,199×each over a served reply's GPU time, as the whole is
- Verify time
- 17.83s/replyn = 3 · 17.66 to 18.07 s/replyverifier wall time
- Proof size
- 18.78MB/replyn = 3 · 18.77 to 18.78 MB/replyevery proof file, as written
- Cost
- proved0.3751USD/replyserved0.0002848USD/replyH2005.345USD/GPU-hour
GPU time, hardware and serving
Serving Meta's FP8 release, vLLM 0.32.0, 4 × H200, full load at 512 concurrent requests · n = 3 timed batches, 1 run (2026-11-11 · 1 (new tab)) · the copy this row proves is under Fidelity
Proving 2 × H200 rented, 1 given to the job and priced, 3 whole sessions of 120 replies
Generation the publisher's integer model, one reply at a time “greedy decoding, one sequence at a time, one GPU” (new tab) · 0.4492GPU-s/reply token0.4457 to 0.4545 GPU-s/reply token · n = 3
Verifying cp_verify on 8 CPUs of an Intel(R) Xeon(R) Platinum 8488C, the prover's host, no GPU visible · 17.66 to 18.07 s/reply, the 3 sessions' means, n = 3 · Lean verifier: not timed in v1.0's sessions (a trial under Formally verified)
Proof size the bytes of the 2 files a proof is released as, as written: response.proof, response.y
Tokens per second 2.027tokens/s per GPU (2.007 to 2.044, n = 3 whole sessions, 1 × H200)
Evidence our runs only, not rerun by anyone else
Ran on Hugging Face Jobs, image flashprove@sha256:12611db7, harness 8512b94d (new tab), adapter aa5f31f9 (new tab)
- Serving GPU time
- 0.1918GPU-s/reply
- Commit GPU time
- 14.76GPU-s/reply14.36 to 15.44 GPU-s/reply
- Proving GPU time
- 7.910GPU-s/reply7.856 to 7.950 GPU-s/reply
- H200 price
- 5.345USD/GPU-houras of 2026-11-08
Serving ladder, 11 concurrencies
| Concurrent requests | Rate |
|---|---|
| 1 | 0.05260replies/GPU-s |
| 2 | 0.1034replies/GPU-s |
| 4 | 0.2053replies/GPU-s |
| 8 | 0.4036replies/GPU-s |
| 16 | 0.7491replies/GPU-s |
| 32 | 1.275replies/GPU-s |
| 64 | 2.010replies/GPU-s |
| 128 | 2.928replies/GPU-s |
| 256 | 3.912replies/GPU-s |
| 512 (full load) | 5.213replies/GPU-s |
| 1,024 | 4.268replies/GPU-s |
Fidelity
- Proved
- 4.84-bit integer weights, fixed-point activationsUD-Q4_K_XL
- Against the proofs
- 11,034 of 11,034 positions357 of 357 where the two replies partwhere a proof states the model's token, FlashProve's code picked it
- Fidelity check
- passedFlashProve29.66tokenspass bar11.15tokensmatched leading tokens, mean
- KL to the original weights
- not measured
The copies, the tests and the sources
Proved UD-Q4_K_XL (new tab) (GGUF), 4.84 bits per weight, computed from its files (new tab). Integer weights, fixed-point activations, as the publisher states: “Model is Llama 4 Maverick's text transformer in integer arithmetic: integer weights, fixed-point activations with fitted power-of-two scales, and lookup tables for the non-linear functions” (new tab)
KL to the original weights: not measured (llama.cpp's KL base is the 16-bit file, which the level's llama.cpp pin cannot run on this rung)
| Replies | Kind | Top-token agreement | Matched leading tokens | KL divergence | |
|---|---|---|---|---|---|
| 8-bit copy (Q8_0) | honest | 96.2%median 96.0 % | 65.71tokens | 0.02669nats/token | |
| FlashProve | this system | 86.4%median 85.4 %95 % interval 85.9 to 86.9 % | 29.66tokensn = 360 replies · above the bar: p < 5.0 × 10−6, the test's floor | 0.1829nats/token | |
FlashProve's top-token agreement and KL divergence: computed by the system's own code on the reference reply, not by the proved computation. What ran: the publisher's integer-model package (IntegerModel.extend), which the publisher says is "by design" the function the prover proves; that it equals cp_prove value by value at full size is not stated. run-flashprove.json, teacher-forced-2.json (new tab) | |||||
| 4-bit copy (Q4_K_M) | honest | 85.5%median 84.2 % | 30.29tokens | 0.2022nats/token | |
| 3-bit copy (Q3_K_S) | damaged, the bar | 71.9%median 68.9 % | 11.15tokens | 0.6366nats/token | |
| 2-bit copy (Q2_K) | damaged | 64.1%median 61.1 % | 5.083tokens | 0.9713nats/token | |
Original: unsloth/Llama-4-Maverick-17B-128E-Instruct at 86c5ecb, 16-bit. It is a copy on the Hub, not Meta's own repository. Fed its own reply, the original's engine picks that reply's token at 99.0 % of positions, which is the figure of the original itself.
Engines: the original and the controls run with it in vLLM, every prompt sent in one call, and the quantized controls in llama.cpp, 16 requests at a time.
Prompts: 512-token windows of 8 texts from www.gutenberg.org (public domain in the United States), with no chat template.
12 tests on top-token agreement, and they set no state
- the original, run again
- FlashProve above it: p = 1.0 · below it: p < 5.0 × 10−6, the test's floor
- 8-bit copy (Q8_0)
- FlashProve above it: p = 1.0 · below it: p < 5.0 × 10−6, the test's floor
- 4-bit copy (Q4_K_M)
- FlashProve above it: p < 5.0 × 10−6, the test's floor · below it: p = 1.0
- 3-bit copy (Q3_K_S)
- FlashProve above it: p < 5.0 × 10−6, the test's floor · below it: p = 1.0
- 2-bit copy (Q2_K)
- FlashProve above it: p < 5.0 × 10−6, the test's floor · below it: p = 1.0
- a smaller model
- FlashProve above it: p < 5.0 × 10−6, the test's floor · below it: p = 1.0
Sessions
3 counted · 5 release groups in each
| Slot | Run | Session | generation | Steady |
|---|---|---|---|---|
| 1 | 2026-11-11 · 1 (new tab) · manifest (new tab) · log (new tab) | 252.3GPU-s/reply | 229.0GPU-s/reply | 236.3GPU-s/reply |
| 2 | 2026-11-12 · 1 (new tab) · manifest (new tab) · log (new tab) | 250.5GPU-s/reply | 228.2GPU-s/reply | noneuneven groups |
| 3 | 2026-11-12 · 2 (new tab) · manifest (new tab) · log (new tab) | 255.1GPU-s/reply | 232.7GPU-s/reply | noneuneven groups |