Decision models, measured

What this page measures

A decision model takes some text and a question with fixed answers, and returns one answer with a probability for each option instead of writing text. Jev made the idea popular; Clef, Kev and others now answer the same kind of request. This page holds our main measurements of them, one question at a time, and grows as new models ship. Each row says when it was measured.

So far all of it comes from one task: 3,080 messages that customers sent to a bank, from the public BANKING77 test split, each to be sorted into one of 77 intents. Every model gets the same question: the message, five labelled examples of similar messages found by a fixed search over the training set, and 78 options, the intents plus none-of-these. We sent the calls one at a time from a Mac mini on a home connection in Berlin.

One task is a narrow base. A model that leads on bank-support intents can trail on your own labels, so read these numbers as a dated measurement on one job. The method section says how they were made, and the files at the foot have every number on this page.

Which one gets the most right?

With five examples, Clef-flash and Clef sorted about a point more messages correctly than Jev, and the interval on each difference stays above zero. GPT-6 Luna and Kev-27B were level with Jev within the noise of 3,080 messages.

Kev-9B and Kev-4B fell behind mostly by answering none-of-these on messages that had a real intent. Kev-4B did it 252 times. With that option set aside, its top real intent was right 93.3% of the time, close to Jev, and Kev-9B's was right 93.6% of the time. A caller who offers an "other" option should expect these two to take it more often than the rest.

Without examples, and with five

Cloudflare launched Clef with a BANKING77 score of 94.20 against Jev's 79.74. Both use the community Decision Index's test: Jev's figure is the Index's own run, Clef's is Cloudflare's run of it. That test asks a different question from ours: the message alone and 77 options described by the intent names, no examples and no none-of-these, scored by macro-F1. We replicated that request on the same messages and got 94.2 for Clef and 80.0 for Jev, so Cloudflare's number holds for the question it asks.

On our question, with five labelled examples, most of the gap closes: Jev scored 13.2 points higher than on the Index's, Clef 0.8 points. The two questions also differ in layout and in offering none-of-these, so not all of that is the examples; on our layout without examples, Jev scored 78.3% in jev-2. If your setup can't show the model any examples, Clef is far ahead on this task. If it can, the models are within a few points of each other, and price and speed decide.

How fast is it from Berlin?

For a call this short, much of the time is the network, so every time here is from Berlin and comes with the network floor to the same host. Jev's median was 235 ms, and a bare request to TypeSafe's host took 171 ms at the start of that run. Clef-flash had the fastest median, 196 ms, but a longer tail than Jev; its slowest calls bunched in the first part of its pass. Clef took 562 ms at the median and GPT-6 Luna 1,097 ms.

Kev-27B and Kev-9B ran on H100s we rented on Modal. The model itself took 110 ms and 35 ms per request by its server's count, and the rest of each call was the path from Berlin to Modal's machine. These are times of our deployment, which we didn't tune, not of a service you can call.

What a thousand decisions cost

Among the hosted models, Kev-4B through OpenRouter was the cheapest at $0.041 per thousand decisions, then Jev at $0.078. Clef-flash cost $0.180 and Clef $0.480.

Price per token is a poor guide here, because each provider counts the same request differently. Jev counted 1,846 input tokens per request and Clef 1,999, while Kev's server counted 975 for the same body. GPT-6 Luna counted 1,066, and 3,032 of its 3,080 calls also paid OpenAI's charge for writing the prompt to its cache, which no call read from.

Running Kev on our own GPU one request at a time cost $0.730 per thousand for Kev-27B. On the shorter requests without examples, sent six at a time, it cost $0.157, still above Jev. At our volume, running the weights yourself buys control of the model and the data path, not a lower price.

How far can you trust the confidence?

Each of these models gives a confidence with its answer, and the practical use is to accept the confident answers automatically and send the rest to a person. That works if the confidence tracks how often the model is right.

Kev-27B's did best: across the range, what it stated was close to how often it was right, with a calibration error of 0.014. Among the models that return probabilities, Clef (0.030) and Jev (0.035) came next. Clef-flash and Kev-4B state numbers well below how often they are right, so a threshold for them has to be set on your own data rather than read off the number. After a simple fit on held-out messages every model's calibration error was 0.021 or lower, and Clef-flash's fell from 0.143 to 0.006.

GPT-6 Luna doesn't return probabilities on this setup. Its confidence is a number it writes in its answer, and it is reasonably well placed, but it is a different kind of number from the others'.

The workload table puts this in a support team's terms. At a budget of 2% wrong among the answers accepted automatically, Clef handled 93.2% of messages alone and Jev 84.4%. Jev's threshold had to sit at 0.990: most of its answers carry a confidence that high, which leaves a threshold little room to separate the right ones from the wrong. GPT-6 Luna's threshold, fixed on the dev messages, let through 2.99% wrong answers on the test, above the budget.

Same answer twice, and over time

Asked the same messages again the same day, Kev-27B, Kev-9B, Clef, Clef-flash and Kev-4B gave identical answers every time. Jev gave the same answer to 199 of 200 and GPT-6 Luna to 197.

Over days, Jev kept the same version string but changed a few answers: 3,075 of 3,080 were the same between 21 September and 30 September, and 3,073 between 30 September and 2 October. Its median time also fell over that week.

Has it seen the test before?

BANKING77 is public, so a model may have trained on it. Kev's model cards list it among their training data. Cloudflare post-trained Clef on an open Qwen3.8 model with its own synthetic data, and doesn't say whether BANKING77 was in that data or in the base model's. We found nothing from TypeSafe on whether Jev saw it.

We asked five of the models, without examples, about 200 messages from the training split and about paraphrases of the same messages; M37's jev-2 asked Jev and GPT-6 Luna the same. A model that had memorised the originals should do much worse on the paraphrases. Every model dropped, but by less than a simple vote of the nearest labelled examples, which can't memorise anything and still drops 15.0 points. So this probe can't tell a little memorisation from the cost of paraphrasing. For Kev it measures something narrower still, since BANKING77 is in its training data.

Would a plain classifier do?

Could a trained classifier do the same job? On this task, with nearly all of the training split, it nearly does. A small embedding model with logistic regression on top, trained on 9,405 labelled messages, got 93.1% right in 9 ms per message on a Mac. Jev got 93.9%.

That comparison favours the classifier, which saw thousands of labelled messages. The live question is how few labelled examples a trained classifier needs before it catches up with a decision model, and that is the next thing we measure.

What we measure next

• How many labelled examples a plain classifier needs to beat Jev and Clef: the same messages, with the classifier trained on a few, then more, examples per intent.
• OpenAI's Decisions API, on the day it opens to us. The code that calls it is already written.
• Latency from other regions, since a Berlin number says little about a caller in Virginia or Singapore.
• Calibration on other kinds of task, including yes-or-no questions.
• Accuracy as the list of options grows.

Each will get its own table here, dated, when it has been measured.

How it was measured

The protocol was written down and committed before any test message was sent to a model, and every model since has run it unchanged. One exception is disclosed in the Kev addendum: before Kev's settings were frozen, 40 test messages went to each Kev model to time the round trip; only the timing was read, and those messages were sent again in the scored pass. Thresholds and the recalibration fits were set on 573 held-out training messages, never on the test. Every model call is kept, with a hash of its request, the full response and its timing.

Accuracy intervals are Wilson intervals. Differences between two models are paired, message by message, with a bootstrap over the messages. The calibration error uses ten equal bands of stated confidence. Costs are the provider's bill where it gives one per call, as OpenRouter does, and the rate card times the tokens it reports where it doesn't. The Kev rows are Modal's bill for the GPU time of each pass.

What this doesn't cover yet: other tasks, other places, many requests at once on the hosted models, images, and long documents. The protocol comes from M37's jev-2 test, which has a video of its own. The per-message file at the foot of this page has every model's answer, confidence and time for each message, so the accuracy, the differences between models and the calibration as returned can all be recomputed from it.