Score
An ordered scale (none / low / medium / high, 1–5, …). The model returns a distribution over levels and the answer is the expected level. The primitive relevance classifiers cannot do.
On Metatext decisions, Jev leads with 0.950 mean accuracy against a 0.500 majority baseline.
Metatext decisions
Every model is zero-shot on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | JevTypeSafe | 0.950 | 0.500 | 1/1 |
| 2 | GLiNER2 largeFastino | 0.950 | 0.500 | 1/1 |
| 3 | GLiNER 2.5 baseFastino | 0.750 | 0.500 | 1/1 |
| 4 | GLiNER 2.5 smallFastino | 0.750 | 0.500 | 1/1 |
| 5 | GLiNER 2.5 multiFastino | 0.650 | 0.500 | 1/1 |
| 6 | GLiNER2 baseFastino | 0.500 | 0.500 | 0/1 |
| 7 | GLiGuard 300MFastino | 0.500 | 0.500 | 0/1 |
| 8 | GLiClass modern large v3.0Knowledgator | 0.450 | 0.500 | 0/1 |
| 9 | GLiNER2 multiFastino | 0.450 | 0.500 | 0/1 |
| 10 | LAYAConvai Innovations | 0.400 | 0.500 | 0/1 |
| 11 | LAYA typed-decisionsConvai Innovations | 0.400 | 0.500 | 0/1 |
| 12 | GLiClass multilang ultraKnowledgator | 0.350 | 0.500 | 0/1 |
| 13 | GLiClass instruct base v1.0Knowledgator | 0.300 | 0.500 | 0/1 |
| 14 | GLiClass instruct large v1.0Knowledgator | 0.300 | 0.500 | 0/1 |
| 15 | Vonwfzyx | 0.300 | 0.500 | 0/1 |
| 16 | LAYA multilingualConvai Innovations | 0.300 | 0.500 | 0/1 |
| 17 | OpenJEVopen-jev | 0.300 | 0.500 | 0/1 |
| 18 | GLiClass edge v3.0Knowledgator | 0.300 | 0.500 | 0/1 |
| 19 | GLiNER multitask v1.0Knowledgator | 0.300 | 0.500 | 0/1 |
| 20 | GLiClass large v3.0Knowledgator | 0.200 | 0.500 | 0/1 |
| 21 | GLiClass base v3.0Knowledgator | 0.200 | 0.500 | 0/1 |
typed-decisions
Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | JevTypeSafe | 0.663 | 0.446 | 7/8 |
| 2 | LAYA typed-decisionsConvai Innovations | 0.593 | 0.446 | 7/8 |
| 3 | GLiClass large v3.0Knowledgator | 0.466 | 0.446 | 4/8 |
| 4 | Vonwfzyx | 0.461 | 0.446 | 3/8 |
| 5 | GLiNER2 largeFastino | 0.445 | 0.446 | 5/8 |
| 6 | GLiClass multilang ultraKnowledgator | 0.381 | 0.446 | 2/8 |
| 7 | GLiNER 2.5 baseFastino | 0.326 | 0.446 | 2/8 |
| 8 | LAYAConvai Innovations | 0.275 | 0.446 | 0/8 |
| 9 | LAYA multilingualConvai Innovations | 0.268 | 0.446 | 0/8 |
The questions
- customer_service.frustration · score · our tasksHow frustrated does the customer appear?calm and neutral · frustrated but civil · very angry
- td_agent_trace_observability.risk · score · LAYA's public suiteHow risky was the agent's behaviour in this trace?Benign: read-only or clearly safe actions. · Low: routine writes within scope. · Moderate: irreversible or out-of-scope actions. · High: destructive, security-relevant, or policy-violating actions.
- td_agent_trace_observability.urgency · score · LAYA's public suiteHow quickly does this trace need attention?No time pressure; can wait indefinitely. · Routine; handle within the normal queue. · Elevated; should be handled within the same week. · Critical; requires action within the same day.
- td_customer_service.churn_risk · score · LAYA's public suiteHow likely is this customer to stop doing business with us because of this interaction?No sign of dissatisfaction. · Mild frustration, but the relationship is intact. · Clearly unhappy; repeat problems or explicit complaints. · Imminent: threatening to cancel, dispute or leave.
- td_customer_service.urgency · score · LAYA's public suiteHow time-sensitive is this conversation?No time pressure; can wait indefinitely. · Routine; handle within the normal queue. · Elevated; should be handled within the same week. · Critical; requires action within the same day.
- td_invoice_processing.discrepancy_severity · score · LAYA's public suiteHow material is any discrepancy between the invoice, the order and the delivery?None: everything reconciles. · Trivial: rounding or a cosmetic difference. · Moderate: a real difference worth confirming. · Material: a large or unexplained difference.
- td_invoice_processing.urgency · score · LAYA's public suiteHow time-sensitive is processing this invoice?No time pressure; can wait indefinitely. · Routine; handle within the normal queue. · Elevated; should be handled within the same week. · Critical; requires action within the same day.
- td_security_incidents.severity · score · LAYA's public suiteHow severe is the potential impact if this alert is real?Negligible: no access to anything sensitive. · Low: limited access, easily reversed. · Moderate: access to internal systems or non-public data. · High: access to production, secrets or customer data. · Critical: active compromise of crown-jewel systems.
- td_security_incidents.urgency · score · LAYA's public suiteHow quickly must a responder act on this alert?No time pressure; can wait indefinitely. · Routine; handle within the normal queue. · Elevated; should be handled within the same week. · Critical; requires action within the same day.
Frequently asked questions
What is a Score question?
An ordered scale (none / low / medium / high, 1–5, …). The model returns a distribution over levels and the answer is the expected level. The primitive relevance classifiers cannot do.
Which model is best at score?
Jev leads on Metatext decisions at 0.950 mean accuracy over 1 questions.
Why do some models score badly on Score?
Score needs an ordered level, not a best match. Relevance classifiers (GLiClass) and NLI heads (Von) return how well each level description matches the text, which collapses to the middle or the most literal level. Only Jev, GLiNER2 and LAYA answer Score above baseline.
Related
A question with a fixed set of named options and a description for each; the model returns a probability per option and the answer is the argmax.
A statement about the state, answered with a probability that it holds. Named after the Jev primitive; the threshold for "yes" is 0.5 unless the caller sets one.
Return the text spans that match each label (person, organization, date, money, …). Jev and the pure decision models never return spans; only the GLiNER family does.