Choice
A question with a fixed set of named options and a description for each; the model returns a probability per option and the answer is the argmax.
On Metatext decisions, Jev leads with 0.870 mean accuracy against a 0.356 majority baseline.
Metatext decisions
Every model is zero-shot on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | JevTypeSafe | 0.870 | 0.356 | 3/3 |
| 2 | GLiClass large v3.0Knowledgator | 0.635 | 0.356 | 3/3 |
| 3 | GLiNER2 largeFastino | 0.620 | 0.356 | 3/3 |
| 4 | GLiNER 2.5 baseFastino | 0.583 | 0.356 | 3/3 |
| 5 | GLiClass multilang ultraKnowledgator | 0.583 | 0.356 | 3/3 |
| 6 | GLiNER2 baseFastino | 0.578 | 0.356 | 2/3 |
| 7 | Vonwfzyx | 0.571 | 0.356 | 3/3 |
| 8 | LAYAConvai Innovations | 0.556 | 0.356 | 2/3 |
| 9 | GLiClass base v3.0Knowledgator | 0.543 | 0.356 | 3/3 |
| 10 | OpenJEVopen-jev | 0.533 | 0.356 | 3/3 |
| 11 | GLiNER 2.5 smallFastino | 0.506 | 0.356 | 3/3 |
| 12 | LAYA typed-decisionsConvai Innovations | 0.501 | 0.356 | 2/3 |
| 13 | GLiNER2 multiFastino | 0.496 | 0.356 | 2/3 |
| 14 | GLiGuard 300MFastino | 0.494 | 0.356 | 2/3 |
| 15 | GLiClass instruct base v1.0Knowledgator | 0.493 | 0.356 | 2/3 |
| 16 | GLiClass modern large v3.0Knowledgator | 0.488 | 0.356 | 2/3 |
| 17 | GLiClass instruct large v1.0Knowledgator | 0.461 | 0.356 | 2/3 |
| 18 | GLiNER 2.5 multiFastino | 0.420 | 0.356 | 2/3 |
| 19 | LAYA multilingualConvai Innovations | 0.389 | 0.356 | 2/3 |
| 20 | GLiNER multitask v1.0Knowledgator | 0.324 | 0.356 | 1/3 |
| 21 | GLiClass edge v3.0Knowledgator | 0.281 | 0.356 | 1/3 |
typed-decisions
Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | JevTypeSafe | 0.743 | 0.492 | 6/6 |
| 2 | LAYA typed-decisionsConvai Innovations | 0.733 | 0.492 | 5/6 |
| 3 | GLiNER2 largeFastino | 0.493 | 0.492 | 1/6 |
| 4 | GLiClass large v3.0Knowledgator | 0.428 | 0.492 | 2/6 |
| 5 | Vonwfzyx | 0.415 | 0.492 | 2/6 |
| 6 | GLiClass multilang ultraKnowledgator | 0.375 | 0.492 | 1/6 |
| 7 | GLiNER 2.5 baseFastino | 0.367 | 0.492 | 1/6 |
| 8 | LAYA multilingualConvai Innovations | 0.295 | 0.492 | 1/6 |
| 9 | LAYAConvai Innovations | 0.288 | 0.492 | 1/6 |
The questions
- agent_traces.outcome · choice · our tasksWhat was the outcome of this agent step?the tool call succeeded · the tool call failed or errored · the wrong tool was used for the goal · the agent repeated itself without progress
- customer_service.department · choice · our tasksWhich team should handle this request?billing and payments · technical support · sales enquiry
- security_incidents.severity · choice · our tasksWhat is the severity of this security incident?critical severity · high severity · medium severity · low severity
- td_agent_trace_observability.action · choice · LAYA's public suiteWhat should the observability system do with this trace?continue · human_review · observe · stop
- td_agent_trace_observability.outcome · choice · LAYA's public suiteHow did this agent run turn out?failure · harmful · partial · success
- td_customer_service.action · choice · LAYA's public suiteWhat should the assistant do next with this conversation?answer_directly · close_no_action · escalate_to_human · execute_refund · request_information
- td_customer_service.category · choice · LAYA's public suiteWhat is this customer conversation primarily about?account · billing · delivery · refund · technical
- td_invoice_processing.disposition · choice · LAYA's public suiteHow should this vendor invoice be dispositioned?approve · hold · manual_review · reject
- td_security_incidents.disposition · choice · LAYA's public suiteHow should this security alert be dispositioned?close_benign · contain · investigate · monitor
Frequently asked questions
What is a Choice question?
A question with a fixed set of named options and a description for each; the model returns a probability per option and the answer is the argmax.
Which model is best at choice?
Jev leads on Metatext decisions at 0.870 mean accuracy over 3 questions.
Related
An ordered scale (none / low / medium / high, 1–5, …). The model returns a distribution over levels and the answer is the expected level. The primitive relevance classifiers cannot do.
A statement about the state, answered with a probability that it holds. Named after the Jev primitive; the threshold for "yes" is 0.5 unless the caller sets one.
Return the text spans that match each label (person, organization, date, money, …). Jev and the pure decision models never return spans; only the GLiNER family does.