Decision Models Need a Decision Layer

A decision model in production does three things. It picks an option, it says “none of these” when no option fits, and it escalates risky requests to a person. Public leaderboards mostly score the first, which has historically led researchers to develop models that would rather hypothesize or confabulate a response than decline to answer. Our research has consistently pointed to one conclusion: improving model outputs is not enough. Reliable automation also requires deciding when to act on them. At Themis AI, we use uncertainty signals from a model’s internal activations to inform those decisions through a model-calibrated decision layer.

We ran Liquid AI’s d1-3B, a leading small open-weight decision model, with TypeSafe’s hosted Jev as the reference point, on three public intent-routing benchmarks that include out-of-scope requests: CLINC150+OOS, HWU64 and BANKING77-OOS. This post reports what changes when the model is scored as a decision rather than as a classifier, and what the Themis Decision Layer adds on top of it.

Scoring decisions instead of answers

The community Decision Index scores CLINC150+OOS by macro-F1 over 151 classes, so “out of scope” counts as one class in 151. In the test set it is 1,000 of 5,500 requests, or 18% of traffic. On that metric d1-3B leads Jev, whereas on plain accuracy, which weights every request the same, the order flips.

Table 1: CLINC150+OOS test set, 5,500 requests. Macro-F1 is the leaderboard’s metric.

The two metrics disagree because the two models fail in different places. On requests that fit one of the 150 options, d1 is right 96.7% of the time against Jev’s 89.7%. On requests that fit nothing, d1 out of the box says “none of these” only 36% of the time, where Jev says it 86% of the time. We can fix most of this by tuning the threshold for choosing the “none” option on labelled validation requests. We follow the same process with Jev.

With each model on its own tuned threshold, d1 makes fewer errors than Jev on all three benchmarks.

Table 2: Errors per 1,000 test requests, every request answered, each model with its own tuned “none” threshold. Hover a reduction for its 95% confidence interval.

Figure 1: accuracy at each share of out-of-scope traffic

The share of out-of-scope traffic varies by deployment, so we also re-tuned each model’s threshold for shares from 0% to 50% and reweighted the test set to match. d1 is more accurate at every share on all three benchmarks, and every interval excludes zero.

Accuracy by share of out-of-scope traffic (higher is better) Jev 1.13d1-3B

CLINC150+OOS

HWU64

BANKING77-OOS

Figure 1: Accuracy as the share of out-of-scope traffic goes from 0% to 50%. Higher is better.

What the Themis Decision Layer adds

Tuning the “none” threshold improves accuracy, but deciding which requests to automate is a separate problem. The Themis Decision Layer adds an uncertainty signal derived from the model’s internal activations to help make that decision.

We compared zero-shot and LoRA-finetuned d1 with and without that signal, alongside hosted Jev. Every system gets the same review step. On HWU64 and BANKING77-OOS, adding the signal increased the share of requests handled automatically while keeping realised error within the 5% target.

Share of traffic handled automatically at a 5% error target (higher is better) Base model onlyAdded by the Themis Decision Layer

HWU64

BANKING77-OOS

Figure 2: Share of traffic handled automatically at a 5% error target, higher is better, for hosted Jev and for d1-3B with and without the uncertainty signal.

How the layer makes the difference

A threshold works with uncertainty the model already expresses. When the model spreads its probability across the options, or gives “none of these” a little weight, the threshold turns that hesitation into a “none” answer. It cannot help when the model is confident and wrong, and that is how the remaining out-of-scope requests fail: the request looks enough like a known option that the model routes it there with high probability. The Themis Decision Layer handles two other critical roles:

• It catches requests the model has no basis to judge. An uncertainty signal that Themis derives from the model’s internals flags inputs the model has not learned to handle, whatever its output probabilities say. This is epistemic uncertainty, the model’s unfamiliarity with the input, rather than the choice between options that its probabilities already express. A validation check decides, per model and task, whether the signal is strong enough to use.
• It decides what goes to a person. A review step, a small model on the decision model’s confidence and the uncertainty signal, chooses which answers to automate, with a threshold set on validation to target a chosen error rate. The rest go to a person.

Because the signal is read from the model’s internals, the layer is built for open-weight models. A hosted API exposes only its outputs, so there is no equivalent for Jev here; d1-3B, whose weights are public, is the model that can carry it. That is also why the comparison in Figure 2 is d1 with and without the layer, with Jev kept as the reference point.

The signal also improves the “none” decision itself. With every request still answered and everything else held fixed, it cut errors by 10 to 18% in five of the six model and benchmark pairs, and on the sixth the validation check switched it off.

Table 3: errors with and without the uncertainty signal

	CLINC150+OOS	HWU64	BANKING77-OOS
Zero-shot d1-3B	73 → 62 (−15%)	223 → 194 (−13%)	127 → 115 (−10%)
Finetuned d1-3B	63 → 63 (no change)	176 → 144 (−18%)	145 → 119 (−18%)

Table 3: Errors per 1,000 test requests, without and with the uncertainty signal. Bold marks the lower error count in each pair.

Nearly all of the gain is out-of-scope requests that had been answered with a known option. The sixth cell is the validation check doing its job. On the finetuned model for CLINC150, which we trained without out-of-scope examples, the signal separated out-of-scope requests only weakly on validation, so the check switched it off. Forcing it on there would have tripled the errors. The check also had little to work with: CLINC150’s validation split holds only 100 out-of-scope requests, too few to establish the separation with confidence, which likely contributed to the switch-off.

Finetuning without out-of-scope examples helped on HWU64 and CLINC150 but made the forced-answer errors on BANKING77-OOS worse, where the held-out intents read like in-scope requests; the review step still benefits from it, as Figure 2 shows.

End to end

Put together, a finetuned d1-3B with the Themis Decision Layer makes fewer errors than hosted Jev on every benchmark, and on HWU64 and BANKING77-OOS it handles about twice as much traffic automatically at the same error target: 60.5% against 35.9% and 69.3% against 32.5%, the brackets in Figure 2. Jev cannot be finetuned, so the like-for-like comparison is zero-shot d1 with the layer, which still automates 1.4 and 1.8 times Jev’s share, 50.0% and 57.1%.

Error rate when every request is answered (lower is better) Jev 1.13Finetuned d1-3B + Themis Decision Layer

Figure 3: Error rate over all test requests, every request answered, hosted Jev against finetuned d1-3B with the Themis Decision Layer. Lower is better.

CLINC150 is the exception for automation, and the reason it is absent from Figure 2. Both Jev and d1 missed the 5% target there: their validation data had 3–8% out-of-scope requests against 18% in test, so thresholds set on it were calibrated for a very different traffic mix. Reweighting validation to the test share closes most of the gap, though both systems still land just above the target. d1 automated more traffic than Jev, but at a higher error rate on what it automated, so we don’t count CLINC150 as a win for either system. A threshold is only as reliable as the validation data it is set on, and in deployment that data should match live traffic.

What benchmarks should measure

In production, a decision model is usually allowed to say “none of these” and to hand hard cases to a person, so benchmarks should score it that way: accuracy with out-of-scope requests at stated, realistic shares; the share of traffic a model can handle at a fixed error rate, with thresholds set on validation and checked on test; and the calibration of the probabilities it serves. On these measures the ranking of the two models changed, and so did what a deployment would gain from each.

This is the same test we described in Before AI Acts: take a strong baseline with its existing controls, add the decision layer, and compare on the same tasks under the same conditions. If you run a decision model where errors are expensive and review is heavy, we would like to run it on your traffic. Get in touch.

Limitations

• The tasks are English intent classification with short requests. HWU64, with 16 intents held out as out of scope, and BANKING77-OOS, with labels corrected against the original BANKING77 release, are our variants of public datasets; CLINC150 uses the official splits.
• Finetuned results average two training seeds, and the uncertainty signal uses one fitted model per setting. Intervals cover test-set sampling, not retraining.
• The review threshold is chosen on validation with a Clopper–Pearson bound. It is a selection rule, not a certified guarantee, and it missed its target on CLINC150.

Further reading

• Catching LLM Hallucinations with Uncertainty: the uncertainty signal applied to a generative model’s outputs.
• Spotting the Unfamiliar: Out-of-Distribution Detection with Uncertainty: the same signal on inputs the model was not trained for, the case closest to “none of these”.
• Before AI Acts: why a decision layer belongs between a model’s output and the action taken on it.
• Azaria and Mitchell, The Internal State of an LLM Knows When It’s Lying (2023): independent evidence that a model’s internal activations carry a signal about whether its output is right.

Methods

• Data. CLINC150+OOS official splits (test: 4,500 in-scope and 1,000 out-of-scope requests). HWU64 with 16 of 63 intents held out as out of scope, chosen by hash (test: 3,000 + 1,000). BANKING77-OOS with labels rebuilt by exact text match to the original BANKING77 release (test: 1,996 in-scope, 2,084 out-of-scope). Validation rows are split into two halves for fitting and threshold selection.
• d1-3B. LiquidAI/d1-3B through Liquid’s own inference code and option readout, every option in one prompt.
• Jev. TypeSafe’s hosted Jev 1.13 through OpenRouter with the identical request, queried 9–16 October 2026.
• Finetuning. LoRA (rank 16) on every linear layer of d1’s decoder blocks, trained for one epoch in d1’s own option format, without out-of-scope examples. On 15% of training rows the correct option is removed and the target becomes “none of these”. Two seeds, averaged. The adapters’ initial values were not controlled by the recorded seed; results are evaluations of the saved models.
• “None” threshold. One threshold per model on its own probability for “none”, chosen for accuracy on labelled validation requests.
• Decision layer. Themis’s uncertainty signal from the model’s internals, subject to its validation check. Review: a logistic model on the model’s confidence, entropy, probability of “none” and the uncertainty signal, cross-fitted on the validation halves, with the threshold from a one-sided 95% Clopper–Pearson bound on validation error.
• Statistics. Paired, stratified bootstrap over test requests (2,000 draws), holding fitted models and thresholds fixed.
• Leaderboard check. When reproducing the Decision Index evaluations, we observed macro-F1 within 0.001 of the board’s entries for d1-3B and Jev on CLINC150 and BANKING77.