typed-decisions leaderboard
typed-decisions is the largest public suite for System One models: four workflows × 100 JSON states × five typed questions. Its gold labels are a teacher distribution's argmax, so the teacher self-agreement ceiling on the card (0.735) is the practical maximum; Jev sits at 0.726.
| # | Model | Decisions | Beats baseline | ECE raw → held-out fitted | p50 / case |
|---|---|---|---|---|---|
| 1 | JevTypeSafehosted | 0.726 | 18/20 | 0.079→ 0.049 | 405 ms† |
| 2 | LAYA typed-decisionsConvai Innovations fine-tuned on this suite | 0.714→ 0.762 | 18/20 | 0.161→ 0.016 | 3.5 s† |
| 3 | GLiNER2 largeFastino | 0.486 | 9/20 | 0.300→ 0.029 | 3.8 s† |
| 4 | Vonwfzyx | 0.461 | 5/20 | 0.249→ 0.027 | 597 ms |
| 5 | GLiClass large v3.0Knowledgator correct payload, no prompt | 0.446 | 7/20 | 0.315→ 0.093 | 977 ms |
| 6 | GLiNER 2.5 baseFastino | 0.398 | 3/20 | 0.415→ 0.059 | 1.3 s† |
| 7 | GLiClass multilang ultraKnowledgator correct payload, no prompt | 0.395 | 3/20 | 0.281→ 0.031 | 672 ms |
| 8 | LAYA multilingualConvai Innovations | 0.345 | 2/20 | 0.322→ 0.073 | 1.2 s† |
| 9 | LAYAConvai Innovations | 0.343 | 2/20 | 0.196→ 0.020 | 3.2 s† |
Ranked by decision accuracy. typed-decisions: LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering. Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.
Same ranking on the other suite: Metatext decisions.
Frequently asked questions
Who made typed-decisions?
It is published under LocalLLaMA on Hugging Face and used by Convai Innovations as the exam for LAYA. We imported the test split unchanged and run every model on it through one harness.
Is LAYA typed-decisions zero-shot on this suite?
No. It was fine-tuned on the suite's train split, so its 0.714 is an in-domain number. It is on the table because the gap it closes is the most useful number on the page.
Related
Leaderboard for the Metatext suite: three hand-written workflows, seven questions, 123 judgments, plus extraction, for Jev and every GLiNER2, GLiClass, LAYA, Von and OpenJEV checkpoint.
An independent benchmark of TypeSafe's Jev against every open System One alternative: decision accuracy, calibration (ECE) and latency on two suites, 2,123 judgments, same harness for every row.
The benchmark for System One models: decision models that answer typed questions with calibrated probabilities. Jev and every open alternative on the same 2,000-decision suite, with accuracy, wins over the majority baseline, ECE and latency.