Field Note 1. Consensus as oracle: TypeSafe’s Jev benchmark as a type specimen

CounterProof Research field note, 16 September 2026, version r7 (corrected 17 September 2026). Drafted by Luuk Joseph Soons with Claude Fable 5.1, one of the two models whose average the benchmark under study treats as truth (see Limits 7). Reviewed refute-first, before publication, by eight outside seats from four model families: Kimi K3 and K2.7 (Moonshot), Grok 4.6 (xAI), DeepSeek v4 pro, and Qwen 3.7 and 3.8 (Alibaba). Every count below was recomputed independently at least six times over the archived files. Seat records and the hashed data archive are held in our reports store and available on request. A plain-language companion is at counterproof.io/notes/typesafe-jev-benchmark-consensus-labels.

Abstract

TypeSafe AI’s Jev is, in its vendor’s words, the “first public System One Model”: it returns only typed decisions (a choice, a score, or a yes/no probability) and never text. Its public benchmark grades nine models against reference labels defined as the mean of GPT-6 Astra and Claude Fable 5.1 at high reasoning. The benchmark site also serves twenty curated per-case files carrying each reference model’s individual answers. We archived those files and counted. On this vendor-curated slice of 20 cases out of 711, the two reference models disagree on the final action in 8 of the 19 cases where both answered (unordered-set comparison of action lists; 6 of 19 under a first-action-only rule) and on 31 substantive splits among 356 question-pairs where both answered; all four cases the vendor labels “all three miss the reference” are cases where the two reference models also disagreed with each other; the security workflow’s second reference substitutes Claude Opus 5, itself a graded model, on “the 88 documents Fable refused or failed”. Jev’s own per-question answers are also published, and on the 31 contested questions it sides with each reference model equally often under argmax (14 / 14 / 3) and nearly so under the vendor’s own expected-value scoring (13 / 15 / 3), a split consistent both with no preferential alignment and with tracking the two labelers’ mean, so the test does not discriminate between them. We use consensus-as-oracle as a name for this grading regime and propose this benchmark as its type specimen. We state what the specimen does and does not license for the agentic-stemmatics programme, name the open problem it adds (a witness whose emission space is fixed by the asker), and declare the authors’ conflict: the drafting model is one of the two reference labelers.

1. The object

Model. Jev (jev-latest; the evaluated build is pinned internally as typesafe:v13_snowy_elephant in the served files). Interface: POST /v1/systemone with a state (string, object or array) and a map of typed questions; primitives Choice (selected option + probability map summing to 1 + confidence), Score (probability per level + confidence; the vendor’s score field is “the probability-weighted answer across the levels; can land between levels”), Noul (probability of yes; no confidence field). The confidence field is documented as “a statistic computed from the probability distribution the answer already gives you”, i.e. a shape statistic, not a probability of correctness. Questions are “evaluated in parallel and in isolation against the same state”. The vendor’s primer states the training path: “Pretrained language models have been adapted in two major ways. TypeSafe adds a third”, meaning reinforcement learning for calibrated decisions (RLCD). The launch post’s FAQ, whose answer bodies live in the page’s Framer hydration payload and appear in no text render (four fetchers reported them absent; a fifth seat found them by searching the raw bytes; extracted and hashed in vendor-pages/launch-post-FAQ-extracted.txt), says two things that bear on provenance. On data: “TypeSafe is primarily a data research lab … We make all the data ourselves. We wouldn’t train on your data even if you asked us to.” On the model: “Jev is neither small nor an LLM, hence being off the intelligence Pareto curve.” The first rules out customer data and names neither a base checkpoint nor a teacher nor whether self-made data carries other models’ answers. The second sits in tension with the primer’s framing of RLCD as an adaptation of pretrained language models; the vendor does not reconcile them, and the trunk remains undisclosed. Reward source is disclosed nowhere we read. The FAQ also states the vendor’s benchmark philosophy: “We deliberately chose not to publish performance against public benchmarks … Disclose the nuance in your evals.” The launch post’s “0% hallucination” figure is, in its own words, “not empirical. Schema matching is guaranteed”, a type-validity property, not a correctness property. Access is waitlist-only.

Benchmark. evals.typesafe.ai: four workflows (Security Incidents 240 cases, Agent Trace Observability 117, Invoice Processing 150, Customer Service 204; 711 total); nine graded models (Jev, Opus 5, Sonnet 5, Haiku 4.5, Sol, Luna, Terra, DeepSeek v4 pro, DeepSeek v4 flash); comparison models run through what the launch post calls “our System One LLM wrapper, which constrains LLMs to output structured decisions compatible with our API”, at provider-default reasoning. Methodology text, verbatim from the served page: “Instead of debating the correctness of the harness and labels, we assume that the code is correct, and measure against the current smartest large models. For this eval, the reference labels are generated via an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking, answering every question in the harness. All other models are evaluated using the provider’s default reasoning settings.” Vendor caveat, launch post: the reference “biases answers towards OpenAI and Anthropic’s models. We likely underestimate the relative performance of our model and DeepSeek’s models.” The scoring function that turns per-question answers into the published accuracy percentages is not in the served files; the consensus row, where published, is titled “mean of Astra and Fable probabilities for every question”, so the reference is defined on probability maps, while the accuracy is presumably an action-level match. We compare actions because that is what the files publish for every model; the mapping from maps to accuracy is the vendor’s, and unseen.

Headline figures (read from the SVG text labels embedded in the served HTML, not from a rendered view): Jev 61.7 / 71.6 / 61.8 / 76.0 on the four workflows, aggregate 67.8%; best comparator Sol 74.1%; Opus 5 73.1%; Terra 67.9; Sonnet 5 67.8; Luna 66.8; DS pro 65.5; DS flash 64.4; Haiku 53.6. Jev trails the per-workflow best on all four (Opus 5 66.2 on security; see §3 on the composition of that workflow’s second reference; Sol on the other three), by 17.3 points on invoices. The same labels carry cost and latency, on which Jev leads the field by one to three orders of magnitude (aggregate $0.0004 · 0.4 s against, for example, Sol $0.0836 · 23.3 s and Opus 5 $0.1761 · 37.8 s); this note concerns the grading regime and says nothing for or against the price-performance claim.

2. Method

Acquisition. curl -sL of the index and four workflow pages, and of the four <workflow>-cases.js?v=<hash> files each page loads. Files are __VIEWER_DATA__({...}) wrappers; strip the wrapper, parse as JSON. Archived with SHA-256 in typesafe-evals-published-cases-2026-09-16/ (this repository), together with the tabulation scripts of the author (analysis2.txt, typesafe-side-on-splits.txt), the Kimi K2.7 seat (kimi-recompute.py, kimi-recompute_output.txt). The Grok seat’s recomputation (rule, per-case list and every figure) is recorded in its RESULT.md (seat record in this repository); the seat removed its own script from the working directory before returning, so only its reported output survives.

Schema. Per workflow: eval.n_cases; eval.examples (five, each with a vendor label); eval.cases[<id>] with models.{opus,sol,typesafe}.decisions[<policy>].actions (list of [ACTION, band]), models.*.nodes[].answers (per-question answers including probability maps), references[] (per reference model: title + final decisions) and reference_answers[<node>][<qid>].sets (two entries where both reference models answered). Two workflows (security, customer service) publish a third references row for the consensus; two (agent-trace, invoice) publish none.

Comparison rule. Final actions compared as unordered sets of ACTION strings; the band element (heavy / light / queue) is dropped, and no case in this slice differs on band given the same actions. Order-sensitive and multiset comparison give identical results (two seats verified). A first-action-only rule gives 6 of 19 instead of 8. A case with a single reference answer (agent_trace_observability/trace_185afd…) is excluded from final-action counts. A question-pair counts as a substantive split only when both sets[i].value are non-null and differ; pairs with one None are reported separately. Jev’s side on a split: Noul at 0.5 (no Jev Noul probability on a split equals 0.5); Choice by argmax of the probability map; Score by argmax and, as the vendor’s own primitive defines it, by expected value rounded to the nearest level; the two rules differ on one of seven Score splits. The reference models’ published Score values equal the argmax of their own maps in all 52 published Score sets; re-deriving them by expected value instead would turn 3 agreements into splits (32, not 31). The published values are the rule of record.

Independence of the count. Three scripts, the author’s, the Kimi K2.7 seat’s and the Grok seat’s, were written independently against the same hashed bytes and agree on every figure in the table below under the stated rules.

3. Results

** The map-level row changes what “visible half” means. Where both reference models published probability maps and their argmax values agree, the maps themselves differ in 151 of 152 pairs. The vendor’s consensus is defined on those maps. So at the level the reference is actually computed, the two labelers almost never agree exactly; the 8/19 and 31/356 figures are disagreement at the level of chosen actions and values, which is the level the published accuracy most plausibly scores. Both levels are reported; neither is a rate.

Two examples, and what each does and does not show. art_T1574.001-2__peer_contained__t0 (vendor label “All three miss the reference”): reference 1 (Astra) ISOLATE HOST; reference 2 REVOKE SESSIONS; consensus row ISOLATE HOST; Opus 5, Sol and Jev all QUARANTINE FILE. This shows a contested label: one labeler did not produce the answer the three models were scored against. It does not show a correct dissent scored wrong: the three models match neither labeler, and would be scored “miss” under either. The case that shows scoring against a contested label going the other way is art_T1546.018-2__change_covers__t2 (vendor label “All three agree”): references split ISOLATE HOST / ESCALATE TIER2, consensus ESCALATE TIER2, all three graded models ESCALATE TIER2, agreement with one labeler against the other, which an action-match scorer would count as correct (the scorer itself is not in the files). Across the slice, wherever a consensus row is published it equals one of the two labelers’ action sets, never a third: the average resolves the labelers’ disagreement by picking a side.

The substitution. Security workflow, references[1].title, verbatim in every case: “Claude Fable 5.1 high, one question per request; Claude Opus 5 high on the 88 documents Fable refused or failed”. The files count cases; the title counts documents; in these files documents ≠ cases (agent-trace: 17 documents for 5 cases; customer service: 18 for 5). No share is computed. The file does not record which documents were substituted, so per-case attribution of reference-2 answers to Fable or Opus 5 is impossible. What follows is one thing: the second labeler for one workflow is a mixture whose composition is undisclosed beyond a document count, and the leaderboard does not surface it. That the substitute model is itself among the nine graded is recorded as a disclosure in Limits 9, not argued here: four seats from four lineages (Kimi, Grok, DeepSeek, Qwen) read successive versions of this paragraph as inviting, by arrangement, a conclusion the text disclaims, and a disclaimer that four independent readers say does not work is not a disclaimer.

4. What the specimen licenses

4.1 Consensus-as-oracle, named. A grading regime in which the reference label is a function (here the mean) of the outputs of other models on the same items, with no independent outcome label. This is not a new theoretical object. It is the paper’s §4 attractor and §5 “thresholding discards minority-true findings” (v1.26 l.400–403) transposed from the reviewer slot to the label slot: the reference carries the labelers’ shared error, which the correlated-errors literature measures as large and increasing with capability (Kim, Garg, Peng & Garg, ICML 2025, arXiv:2506.07962, measured on two leaderboards and a résumé-screening task, with 60% same-wrong-answer agreement on one leaderboard and 42% on the other per the live paper’s v1.30 wording, https://research.counterproof.io/agentic-stemmatics.html, archived with hash in the vendor-pages/ folder of the data archive; transfer to workflow benchmarks is our inference). One dependence to state: the vendor’s accuracy function is unseen (§1); if it credits probability-map proximity rather than action match, “scores correct / scores wrong” softens to “scores nearer / scores farther”, and the argument runs the same way with less bite. A candidate reproducing the shared error scores correct; a candidate correctly dissenting scores wrong. What is new is the specimen: a shipped vendor benchmark whose visible half, the labelers disagreeing with each other, resolved silently by the average, is countable in published per-case files, and whose graded model is a witness with an asker-fixed emission alphabet. The invisible half, both labelers wrong together, is undefined without an independent outcome; we state that as a limit of the evidence, not as an argument.

4.2 What it adds to the corpus. The paper (v1.26 pin; live v1.30) names the brief as a correlation channel (§11.6), the attractor (§4), direct and multi-teacher distillation (§6, §7.1 “contaminatio formalised”), unanimous silence (§11.7), and closed-set attribution (§4: Sun et al.’s five-way classifier; the contrast with reference-based tests at l.365–368). It has no dedicated passage on (a) a benchmark or evaluation label source as an ancestor-type object, (b) a witness whose emission space is fixed by the asker, or (c) a cheap model as pre-filter (three absences grep-confirmed by two outside lineages). A fourth absence claimed in r1, “closed-set behavioural fingerprints”, was too broad: §4 is a dedicated passage on closed-set attribution from text; what is absent is the narrower object, fingerprints on a typed witness’s probability maps. Items (a) and (b) are the candidate additions. Item (b) is our joint reading of §11.6 and §11.7 (the option set is the brief, and nothing unoffered is ever emitted), not a sentence either passage contains. A typed witness can signal misfit (a flat map) but cannot name it; so the entropy of the returned map, not the argmax, is the instrument for detecting an incomplete option set.

4.3 What it does not license. Nothing here is evidence about Jev’s ancestry. The idiosyncrasy channel (Sun et al., ICML 2025, arXiv:2502.12150, word-level distributions) is closed for a model that emits no words; engineered markers and reference-based teacher tests (Rawat et al., arXiv:2607.09692, which require an earlier-generation checkpoint of the same lineage) need access an outsider lacks. On the corpus’s own rule (§13: “earned for engineered markers and reference-based teacher tests only”), typed-decision ancestry is program, not result. The distillation hypothesis, that RLCD’s reward came from a frontier average, making Jev a descendant of two panel lineages rather than a third, is held at possible: mid-pack agreement is weak evidence against (capability and correlation confounded), and the 14/14 (or 13/15) split is non-discriminating: a model with no favourite and a model trained to follow the two labelers’ mean would both split evenly on questions where the labelers part company. Falsifiers: a disclosed non-frontier label source; or a comparison of how close Jev’s probability maps sit to the labelers’ mean, set against competent models not trained on either labeler, across all 711 cases under a frozen rule. Proximity alone does not show descent: any competent model tends to fall between two competent labelers.

4.4 Consequence for review panels. A panel gate that counts vendors as lineages should not count a typed-decision model graded against other seats’ consensus as an independent lineage until its ancestry has been tested by one of the two earned instruments. That follows from §5 and §13 without new data. “UNRESOLVED” is our house label for that state, not a corpus result, and the files here do not license the stronger default that such a model is a descendant. The failure mode the rule guards is silent: an untested seat reads as an added independent one.

5. Limits

1. Twenty curated cases of 711, vendor-selected to show five situations; no rate estimate follows, and the abstract’s counts are counts on that slice.
2. All figures are the vendor’s own measurements; nothing was re-run; the scoring function from per-question answers to published accuracy is not in the files.
3. Per-case files name three graded models only; the nine-model table comes from the pages’ SVG labels.
4. Vendor documentation and launch-post quotations were live-verified by four seats on 16 Sep and are now archived with SHA-256 in typesafe-evals-published-cases-2026-09-16/vendor-pages/; before that archive existed they rested on live fetches only.
5. The single-reference case and the two workflows without a consensus row limit which counts are re-derivable; where the vendor’s label says “miss the reference” and no consensus row exists, the reference is not visible in the files, and the label cannot be re-derived.
6. The 14/14/3 split is rule-dependent (13/15/3 under the vendor’s expected-value Score); the 8/19 is rule-dependent (6/19 first-action-only). The stated rules are the ones of record.
7. Two of the drafting parties (Fable 5.1 via this author’s tooling; an OpenAI harness for a consulted analysis) are the benchmark’s labelers. Outside seats to date: Kimi K3 (assessment), Kimi K2.7 (article r1), Grok 4.6 (this note, r1), DeepSeek v4 pro (article r4), Kimi K2.7 (this note r2: zero refutations), Kimi K3 (article r4: the FAQ find), Qwen3.8 (this note r5, CSO-run: zero count refutations; article, pending). The sentences have not been seat-read as written.
8. Corpus line numbers cited are v1.26; the deposited paper is v1.29 (DOI 10.5281/zenodo.22790100), the live HTML is labelled v1.30; every § cited was verified present in v1.30 on 16 Sep, and the Kim et al. wording above follows v1.30.
9. Disclosure, drawing nothing from it: Claude Opus 5, the substitute for Fable 5.1 on “the 88 documents Fable refused or failed” in the security workflow, is also one of the nine graded models. The files do not identify the substituted documents; a substituted answer still has to win the mean against Astra before it becomes the label; no inference about any model’s score follows from this fact, and none is drawn.

6. Proposed open-problem entry (agenticstemmatics.org, appended as problem 7)

7. The witness with a fixed emission space. A model that returns only typed decisions over an asker-supplied option set cannot raise an unoffered reading: the option set is the whole brief (§11.6) and nothing unoffered is ever emitted (§11.7). Its probability map can signal misfit but cannot name it. Two instruments are needed. First, a per-item test of whether a flat or renormalised map tracks genuinely incomplete option sets, against oracle-labelled omissions, a calibration and out-of-distribution measurement on a closed action set that does not depend on any vendor. Second, a comparison of how closely the witness’s probability maps sit to the graders’ mean, set against competent models known not to be trained on those graders, over a full benchmark under a frozen rule rather than a curated slice. Proximity alone does not show descent: any competent model tends to fall between two competent graders. The comparison measures behavioural resemblance; training ancestry still needs engineered markers or reference-based teacher tests. A negative result on either changes what a panel may claim about such a seat. (§4, §7.2, §11.6–7; Field Note 1.)

Correction, 17 September 2026. Section 6 previously described the second instrument as a per-question error-correlation measure, which disagreed with the open problem as published on agenticstemmatics.org, and section 4.3 offered proximity to the labelers’ mean as a test that discriminates descent. Proximity alone does not show descent. Both passages now require comparison with competent models not trained on the graders. Found by a reviewer seat from OpenAI’s Codex while reviewing our evaluation plan for Jev; the substance was confirmed by a Moonshot Kimi seat.

Sources

TypeSafe: homepage (source of the quoted “first public System One Model”); launch post; docs introduction, introduction/machine-learning-primer, confidence, concepts/system-one, api.md, primitives/choice.md; evals.typesafe.ai index, four workflow pages, four *-cases.js (archived, hashed). Kim, Garg, Peng & Garg, ICML 2025, arXiv:2506.07962. Sun, Yin, Xu, Kolter & Liu, ICML 2025, arXiv:2502.12150. Rawat, Chen, Anand, Duan, Rotsted & Min, arXiv:2607.09692. Soons, Agentic Stemmatics v1.29, doi:10.5281/zenodo.22790100 (live HTML v1.30). Internal: TYPESAFE-JEV-ASSESSMENT-2026-09-16.md (r4), its two seat records, the Grok seat record on this note, and the archive folder named above.

CounterProof Research Ltd is a small practice that reviews AI-generated code and AI-assisted review processes for organisations where a wrong answer is expensive. We disclose, before anything else, that we have a commercial interest in the adoption of outcome-labelled evaluation, the alternative to the grading regime this note describes, and that the model used to draft this note is one of the two whose average the benchmark under study treats as truth.

Cite as: Soons, L. J. (2026). Field Note 1. Consensus as oracle: TypeSafe's Jev benchmark as a type specimen. CounterProof Research, 16 September 2026, r7. Companion to Agentic Stemmatics, doi:10.5281/zenodo.22030516.