Jev vs GLiClass
GLiClass was the row that looked worst on the first pass and turned out to be a payload bug: it prepends the prompt to the text and scores relevance, not truth. With criteria folded into each label and one hypothesis per Noul question it is above every GLiNER checkpoint except the top two.
undisclosed · undisclosed · Proprietary (hosted API)
DeBERTa-v3-large · 439M · Apache 2.0
Metatext decisions
Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat.
| Metric | Jev | GLiClass large v3.0 |
|---|---|---|
| Decision accuracy | 0.863 | 0.642 |
| Questions over baseline | 6/7 | 5/7 |
| ECE raw → held-out fitted | 0.093 | 0.250 |
| p50 per case | 441 ms | 977 ms |
| Extraction F1 | n/a | n/a |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | GLiClass large v3.0 |
|---|---|---|---|---|---|
| agent_traces | outcome | choice | 0.357 | 0.786 | 0.429 |
| agent_traces | needs_human | noul | 0.714 | 0.714 | 0.714 |
| customer_service | department | choice | 0.400 | 0.950 | 0.850 |
| customer_service | refund_requested | noul | 0.700 | 0.900 | 0.950 |
| customer_service | frustration | score | 0.500 | 0.950 | 0.200 |
| security_incidents | severity | choice | 0.312 | 0.875 | 0.625 |
| security_incidents | actively_exploited | noul | 0.688 | 0.938 | 0.750 |
typed-decisions
LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering.
| Metric | Jev | GLiClass large v3.0 |
|---|---|---|
| Decision accuracy | 0.726 | 0.446 |
| Questions over baseline | 18/20 | 7/20 |
| ECE raw → held-out fitted | 0.079 → 0.049 | 0.315 → 0.093 |
| p50 per case | 405 ms | 977 ms |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | GLiClass large v3.0 |
|---|---|---|---|---|---|
| td_agent_trace_observability | action | choice | 0.360 | 0.620 | 0.310 |
| td_agent_trace_observability | needs_review | noul | 0.550 | 0.600 | 0.560 |
| td_agent_trace_observability | outcome | choice | 0.500 | 0.800 | 0.510 |
| td_agent_trace_observability | risk | score | 0.340 | 0.630 | 0.340 |
| td_agent_trace_observability | urgency | score | 0.370 | 0.360 | 0.350 |
| td_customer_service | action | choice | 0.520 | 0.670 | 0.240 |
| td_customer_service | category | choice | 0.380 | 0.930 | 0.730 |
| td_customer_service | churn_risk | score | 0.470 | 0.860 | 0.560 |
| td_customer_service | needs_human | noul | 0.670 | 0.650 | 0.330 |
| td_customer_service | urgency | score | 0.420 | 0.840 | 0.740 |
| td_invoice_processing | discrepancy_severity | score | 0.470 | 0.570 | 0.150 |
| td_invoice_processing | disposition | choice | 0.450 | 0.650 | 0.450 |
| td_invoice_processing | duplicate | noul | 0.890 | 0.920 | 0.110 |
| td_invoice_processing | matches_order | noul | 0.540 | 0.960 | 0.540 |
| td_invoice_processing | urgency | score | 0.510 | 0.680 | 0.690 |
| td_security_incidents | credential_compromise | noul | 0.550 | 0.820 | 0.400 |
| td_security_incidents | disposition | choice | 0.740 | 0.790 | 0.330 |
| td_security_incidents | severity | score | 0.490 | 0.790 | 0.510 |
| td_security_incidents | true_positive | noul | 0.730 | 0.810 | 0.680 |
| td_security_incidents | urgency | score | 0.500 | 0.570 | 0.390 |
Verdict
Jev on every decision question. GLiClass at 0.642 (best config) on the Metatext suite and 0.446 on typed-decisions, with one structural gap: it cannot answer Score at all, because its output is a relevance, not an ordered level.
Frequently asked questions
Why do the GLiClass rows have variants?
Because the payload changes the answer by up to +0.25. The plain rows use label names only; -desc folds the Jev criteria into each label; -desc-nli also presents Noul as one hypothesis and drops the prompt. The last is the correct mapping.
Related
Fastino's GLiNER2 large against Knowledgator's GLiClass large v3.0 on decisions, calibration, latency and extraction.
Every open-weights alternative to TypeSafe's Jev, ranked by measured decision accuracy on the 2,000-decision typed-decisions suite: LAYA, Von, GLiNER2, GLiClass, OpenJEV, plus the Jev-shaped checkpoints not yet run.