Reported by the authors or by third parties, on their own suites. Not comparable to the leaderboard above; attributed so you can check them.
VendorTypeSafe · 2026-09-14
Measured on: TypeSafe's four workflow evals (711 cases)
- End-to-end latency
- 70–500 ms (vs 3–329 s for frontier LLMs)
- Price
- $0.042 per 1M input tokens; output free
- Headline claim
- 193.6× faster, 444.6× cheaper on the workflow evals
- Schema errors
- 0% by construction (not empirical)
Vendor. Harness not released; reference labels are the average judgment of two frontier LLMs, not ground truth.
No calibration (ECE) figure is published anywhere by TypeSafe.
VendorTypeSafe
Measured on: Four workflows, 711 cases, consensus labels from GPT-6 Astra + Claude Fable 5.1
- Accuracy (aggregate)
- 67.8% · $0.0004/case · 0.4 s
- Security incidents (n=240)
- 61.7% vs best LLM 66.2%
- Agent trace observability (n=117)
- 71.6% vs 76.6%
- Invoice processing (n=150)
- 61.8% vs 79.1%
- Customer service (n=204)
- 76.0% vs 78.3%
Vendor dashboard. Jev loses the accuracy column and wins cost and latency; the page notes possible harness bias. Per-workflow cells transcribed by Kingy AI.
VendorTypeSafe · 2026-07-31
Measured on: 182 Hermes skills, 488 requests, agent = Claude Haiku 4.5, model = jev-1.12
- Agent loads the wrong skill
- 16.8% alone → 7.3% with Jev suggestion → 2.5% given the answer
- Loads a skill when none fits
- 9.8% → 4.0% → 1.2%
Vendor cookbook with the harness described; not independently reproduced.
VendorTypeSafe · 2026-08
Measured on: CLERC legal retrieval (40 queries); GDPR article (13 questions); 60 SEC reports × 75 industries
- Re-ranking BM25 top-30 (CLERC)
- top-1 5% → 18%, top-10 38% → 62%
- Batching 13 questions in one request
- 12.2× cheaper, 10.0× faster than one at a time
- Confidence gating (SEC industry, 75 classes)
- forced 65%; answers with p ≥ 0.9: 90% (27/30)
Vendor cookbooks; index verified, per-page figures partly via third-party write-ups.
VendorOpenRouter · 2026-09-18
Measured on: Listing
- Price
- $0.042/M input, $0 output
- Context
- 32K (64k total with a 32k state budget per systemonemodels.org)
- Limits
- 255 Choice options, Score ≤ 10 levels
Marketplace listing; price and context window only.
Third partyLiteLLM · 2026-09-18
Measured on: 80 authored routing cases × 3 runs
- Routing accuracy
- 95.0% (228/240) vs Claude Haiku 4.5 73.75%
- Latency p50
- 127 ms vs 688 ms (5.4×)
- Cost
- $0.0077 vs $0.1985 (96% cheaper)
Third-party, config given, cases not public.
Third partynibzard · 2026-09-18
Measured on: Banking77 (77-way), spam, code-word up to 512 options, permuted option order
- Banking77 / spam / permuted
- 76.3% / 93.0% / 76.7%
- ECE
- 0.246 (worst of the field; LLMs 0.039–0.122)
- Option-order flip rate
- 13%
- Latency p50
- 264–276 ms; 400 error at ≥ 256 options
Third-party, full code, logs and frozen items public. The strongest independent calibration measurement of Jev.
Third partyickma2311 · 2026-09-18
Measured on: Banking77 (n=208), CLINC150 (n=200), pre-registered
- Banking77
- 0.832 [0.779, 0.880] vs GPT-5.6 Terra 0.875; supervised bge-small + LR 0.933
- CLINC150
- 0.870 vs Terra 0.915
- Overconfidence
- confidence 1.0 on 102/200 items, 6 of them wrong
Third-party, pre-registered, code public.
Third partySimon Smith · 2026-09-18
Measured on: Banking77 full test (3,080), 24 BM25 few-shot examples in the state
- Accuracy with examples in state
- 92.40% vs fine-tuned BERT 93.66%
- Zero-shot screening (n=154)
- 79.22%
- Cost / time
- $0.44, 6m51s for 3,080 cases
Third-party, code public.
Third partyakhilaaa3
Measured on: Two sets × 80 states / 293 questions, writer-labelled
- Medium set
- 88.05%, ECE 0.042, $0.00017/state (Gemini 3.8 Flash 99.3%, Sonnet 5 98.6%)
- Hard set
- 68.26%, ECE 0.089 (LLMs 89–95%)
Third-party, dataset public.
Third partyLargitData · 2026-09-20
Measured on: 100 multi-turn RAG-routing decisions (20 conversations × 5), multilingual
- All four sub-decisions correct
- 61.4% (p50 749 ms)
Third-party, per-item logs on GitHub.
Third partyLuni
Measured on: PhishNChips (2,000 emails) and typed-decisions
- Phishing accuracy / ECE / AUROC
- 0.626 / 0.154 / 0.689 (Haiku 4.5 0.813)
Third-party scripts; the Jev rows are quoted from elsewhere, not measured here.
Third partyWotAI · 2026-09-18
Measured on: 150 prose-voice passages × 16 models, plus two more tasks
- Accuracy / ECE / p50
- 66.0% / 0.121 / 455 ms
- Accuracy in the 0.95–1.0 confidence bin
- 88.5% (n=122)
Third-party, data not public.
Third partyBen Greenberg · 2026-09-18
Measured on: 102 submissions × 3 runs
- Accuracy (Choice + 4 Nouls)
- 100.0%; Choice alone 99.3%; Sonnet 5 99.0%
- ECE
- 0.037 vs Sonnet 5 0.058
- p50 / cost per 10k
- 378 ms / $2.27 vs 3,554 ms / $129.74
Third-party, private data.
Third partyLangChain · 2026-09-20
Measured on: 5 agent runs × 100 repetitions, human oracle
- does_pass agreement
- 500/500; GPT-5.6 Terra 99.8%; Sonnet 4.6 80.0%
- Run-to-run variance
- 92–913× lower than the LLM judges
- Latency / cost
- 0.44 s, $0.00035 per call
Third-party, small n.
Third partynavinpai
Measured on: 400 intent, 400 MMLU-Pro, 100 relational questions
- Intent / MMLU-Pro / relational
- 99.75% / 79.75% / 0.00%
Third-party; method described, code not confirmed.
Third partyJared Palmer
Measured on: decision-v4 and transfer-v4 frozen dev sets
- In-distribution / out-of-domain
- 0.845 / 0.855
Independent measurement of Jev via Vercel AI Gateway; repo public.
Third partyAman Kumar · 2026-09-18
Measured on: ~16k calls across four public sets and a private pipeline
- Page-type gates
- 95.8–98.0% outcome agreement
- Spam (n≈18.5k)
- 98.3% zero-shot ≈ TF-IDF logistic regression 98.4%
- Phishing
- 62.6% vs Haiku 81.3%; 95% with five decomposed Nouls
Third-party, private data.
Third partyLangfuse (citing Good Start Labs) · 2026-09-18
Measured on: 6,003 rubric checks
- Agreement with Claude Fable 5.1 verdict
- 91.5%
- Cost per 1M graded
- $160 vs $33,000 (Fable) / $400 (Luna)
Secondary source.
Third partySam Reghenzi · 2026-09-21
Measured on: 50 products, taxonomy assignment
- Mean latency
- 1.38 s vs 9.62 s agentic loop
- LLM calls / tokens
- −56% calls, −36% tokens
- Response time vs question count
- flat 0.32–0.39 s for 1–7 questions
Third-party, private data.
Third party0xNatoshi
Measured on: 237 replayed coding-agent turns
- Simulated token spend
- ≈ −60% vs routing everything to the frontier model
- Router overhead
- ~$0.00003 and 0.6 s per turn
Historical simulation; README says quota saved was not measured.
Third partyunverifiedArcher Hume · 2026-09-17
Measured on: ~10k probe calls
- MMLU-Pro / ECE (claimed)
- 84.6% / 0.031
- Throughput (claimed)
- ~30k tokens in ~160 ms; up to 1,500 questions per request
Page fetched; the figures came from a search snippet, not the visible body. Treat as unverified.
Model cardConvai Innovations
Measured on: typed-decisions test split (400 cases, 2,000 decisions)
- Accuracy / ECE / score MAE (quoted)
- 0.727 / 0.144 / 0.391
Author-reported; the Jev row is quoted as "published", not measured by them. Trained on the suite's own train split.
Model cardConvai Innovations
Measured on: AG News, DAIR Emotion, Banking77
- AG News / Emotion / Banking77 (quoted)
- 0.910 / 0.480 / 0.870
Reproducible for LAYA; the Jev rows are second-hand (nibzard and others) and used 72 Banking77 labels vs 77.
Model cardwfzyx
Measured on: NLI held-out mixture; jabr classifier-benchmark v1 (78 cases) and v2 (869 cases)
- jabr v2 macro (49 tasks)
- 96.6%; Von 72.0%; GLiNER2 large 68.4%; LAYA 58.3%
Author-reported. The "91.23%" is an NLI held-out set, not a Jev suite. The comparison script in the repository calls only Von and LAYA, so the Jev rows are not produced by its code.
An "88.30% Jev" table quoted in secondary sources could not be found in any current revision of the README or card.