Metatext decisions leaderboard

The first exam, hand-written to mirror the domains TypeSafe uses in its own docs. Small enough that one flipped judgment moves a question by 0.07, so treat ±0.1 as a tie and read the ordering off typed-decisions.

Leads on decision accuracy: Jev — 0.863 on Metatext decisions

#	Model	Decisions	Beats baseline	ECE raw → held-out fitted	p50 / case	Extraction F1
1	JevTypeSafehosted	0.863	6/7	0.093	441 ms†	n/a
2	GLiNER2 largeFastino	0.716	6/7	0.261	627 ms†	0.968
3	GLiNER 2.5 baseFastino	0.643	5/7	0.251	332 ms†	0.968
4	GLiClass large v3.0Knowledgator correct payload, no prompt	0.642	5/7	0.250	977 ms	n/a
5	GLiNER2 baseFastino	0.631	3/7	0.350	190 ms†	0.935
6	GLiClass multilang ultraKnowledgator criteria + single-hypothesis Noul (correct payload)	0.628	4/7	0.185	672 ms	n/a
7	GLiClass base v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.606	5/7	0.146	300 ms	n/a
8	GLiClass modern large v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.602	4/7	0.234	572 ms	n/a
9	GLiNER 2.5 smallFastino	0.574	4/7	0.172	132 ms†	0.812
10	GLiClass instruct base v1.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.559	3/7	0.390	323 ms	n/a
11	GLiClass instruct large v1.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.545	3/7	0.308	1.0 s	n/a
12	GLiGuard 300MFastino	0.537	3/7	0.230	155 ms†	0.000
13	LAYAConvai Innovations	0.520	3/7	0.064	378 ms†	n/a
14	Vonwfzyx	0.509	4/7	0.234→ 0.122	597 ms	n/a
15	GLiNER 2.5 multiFastino	0.494	4/7	0.244	346 ms†	0.885
16	LAYA typed-decisionsConvai Innovations	0.475	3/7	0.131	375 ms†	n/a
17	GLiNER2 multiFastino	0.464	2/7	0.354	202 ms†	1.000
18	LAYA multilingualConvai Innovations	0.431	2/7	0.284	149 ms†	n/a
19	OpenJEVopen-jev	0.427	3/7	0.210	50 s†	n/a
20	GLiClass edge v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.418	2/7	0.287	41 ms	n/a
21	GLiNER multitask v1.0Knowledgator class names as labels	0.378	1/7	0.223	1.8 s	n/a

Ranked by decision accuracy. Metatext decisions: Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat. Every model is zero-shot on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.

Same ranking on the other suite: typed-decisions.

Frequently asked questions

Why is this suite so small?

Every judgment was written and labelled by hand. It exists to sanity-check the public suite and to score extraction, which typed-decisions does not have.

typed-decisions leaderboard

Leaderboard for LAYA's public typed-decisions suite: Jev, LAYA, Von, GLiNER2 and GLiClass on 2,000 decisions across four workflows, with per-question detail and calibration.

System One scorecard

One table, every System One model: Jev, LAYA (three checkpoints), Von, OpenJEV, seven GLiNER2 and seven GLiClass checkpoints. Decision accuracy, calibration and latency on the Metatext suite, extraction F1 where the model has a span head.

Frequently asked questions

Related