The Newest Decision Models Still Need a Decision Layer
We ran Liquid AI’s new d1.1 and d1-ultra and TypeSafe’s Jev 1.14 through the intent-routing tests from our last post. An open 3B model, finetuned and given the Themis Decision Layer, still came out ahead.
On this page
In our last post, Decision Models Need a Decision Layer, a finetuned d1-3B with the Themis Decision Layer made fewer errors than TypeSafe’s hosted Jev 1.13 on three intent-routing benchmarks and handled about twice as much traffic automatically at the same error target. Decision models are improving quickly, so we have run Liquid AI’s new hosted models, d1.1 and d1-ultra, and TypeSafe’s Jev 1.14 through the same evaluation. The hosted models get the same “none” threshold and review step as d1-3B, but not the layer: it reads its uncertainty signal from a model’s internal activations, which hosted APIs do not expose. The d1-3B results are the first post’s.
Finetuned d1-3B with the layer still handled more traffic automatically than all four hosted models on HWU64 and BANKING77-OOS while meeting the 5% error target.
HWU64
BANKING77-OOS
Figure 1: Share of traffic handled automatically at a 5% error target on HWU64 and BANKING77-OOS, for the four hosted models and for d1-3B with and without the layer.
At the 5% target, that means 22–31% fewer requests sent to a person than with the best hosted model on each benchmark, from an open 3B model that can run on your own hardware and be finetuned on your own data.
CLINC150+OOS is not in Figure 1: only Jev 1.14 met the 5% target there (Table 1), for the reason given in the first post.
Table 1: share automated at a 5% target for every system, including CLINC150+OOS
| System | CLINC150+OOS | HWU64 | BANKING77-OOS |
|---|---|---|---|
| Jev 1.13 (hosted) | 83.4% (5.2% ✕) | 35.9% (4.3%) | 32.5% (1.4%) |
| Jev 1.14 (hosted) | 85.6% (4.6%) | 36.4% (3.5%) | 33.3% (1.8%) |
| d1.1 (hosted) | 99.1% (6.7% ✕) | 40.2% (3.2%) | 55.6% (2.3%) |
| d1-ultra (hosted) | 98.7% (6.9% ✕) | 49.6% (4.2%) | 54.3% (2.7%) |
| d1-3B, zero-shot | 100.0% (7.2% ✕) | 33.9% (3.8%) | 56.0% (3.4%) |
| d1-3B, zero-shot + layer | 99.9% (7.2% ✕) | 50.0% (4.5%) | 57.1% (2.9%) |
| d1-3B, finetuned | 100.0% (6.3% ✕) | 51.6% (4.1%) | 63.9% (5.0%) |
| d1-3B, finetuned + layer | 100.0% (6.3% ✕) | 60.5% (4.9%) | 69.3% (3.5%) |
Table 1: Share of traffic handled automatically at a 5% error target, with realised test error in brackets; ✕ = realised error above the target (rounded to one decimal place).
With every request answered, finetuned d1-3B with the layer made fewer errors than d1.1, the hosted model with the fewest errors, on CLINC150+OOS and HWU64, and was level within noise on BANKING77-OOS. Against Jev 1.13, Jev 1.14 and d1-ultra, it made 16% to 44% fewer errors on all three benchmarks.
Figure 2: Error rate over all test requests, every request answered, d1.1 against finetuned d1-3B with the Themis Decision Layer. Lower is better.
d1.1 is real progress
d1.1 is a much better decision model than Jev. With each model given its own tuned “none” threshold, it made 22–32% fewer errors than Jev 1.14 on all three benchmarks and fewer than d1-ultra on all three, and it served the best-calibrated probabilities of the four hosted models on HWU64 and BANKING77-OOS.
Table 2: errors of the four hosted models
| CLINC150+OOS | HWU64 | BANKING77-OOS | |
|---|---|---|---|
| Jev 1.13 | 110 | 258 | 170 |
| Jev 1.14 | 99 | 253 | 162 |
| d1.1 | 72 | 172 | 126 |
| d1-ultra | 75 | 188 | 150 |
| Fewer errors with d1.1 than Jev 1.14 | 28% | 32% | 22% |
Table 2: Errors per 1,000 test requests for the hosted models, every request answered, each with its own tuned “none” threshold. Hover a reduction for its 95% confidence interval. d1.1’s expected calibration error on HWU64 and BANKING77-OOS was 0.024–0.028, against 0.070–0.102 for the other hosted models there.
It takes both steps
Against d1.1, which step matters depends on the task. On HWU64, neither finetuning nor the layer was enough alone: finetuned d1-3B without the layer was level with d1.1 (176 against 172 errors per 1,000), zero-shot d1-3B with the layer was above it (194), and only the two together made fewer (144). On BANKING77-OOS, finetuning alone made errors worse (145, against 127 out of the box); the layer brought them back to level with d1.1 (119) and raised automation from 57.1% with the layer alone to 69.3%. On CLINC150+OOS, either step alone was enough. So we recommend both: finetune the model, and add the decision layer.
Table 3: each d1-3B variant against d1.1
| CLINC150+OOS | HWU64 | BANKING77-OOS | |
|---|---|---|---|
| d1.1 (hosted) | 72 | 172 | 126 |
| d1-3B, zero-shot | 73 (+1%) | 223 (+30%) | 127 (+1%) |
| d1-3B, zero-shot + layer | 62 (−14%) | 194 (+13%) | 115 (−9%) |
| d1-3B, finetuned | 63 (−12%) | 176 (+2%) | 145 (+15%) |
| d1-3B, finetuned + layer | 63 (−12%) | 144 (−16%) | 119 (−5%) |
Table 3: Errors per 1,000 test requests for each d1-3B variant, with the change against d1.1 in brackets. Green: fewer errors than d1.1, red: more, with the 95% interval excluding zero; plain: within noise. Hover a cell for its interval. On CLINC150+OOS the validation check switched the signal off for the finetuned model, so the two finetuned rows match there.
If you run a decision model where errors are expensive and review is heavy, we would like to run it on your traffic. Get in touch.
Methods
- As in the first post. Data, d1-3B and its finetuning, the “none” threshold, the decision layer and review step, statistics and limitations are described in the first post’s Methods and Limitations.
- Hosted models. TypeSafe’s Jev 1.13 and Jev 1.14 and Liquid AI’s d1-ultra through OpenRouter; Liquid AI’s d1.1 through Liquid AI’s API, each with the identical request. Hosted models are updated over time; the results are for the versions we queried in October 2026.
