Does Cloudflare's Clef really beat Jev?

This is the video written out, with every figure in full, each linked to its source in the table below.

Cloudflare's claim▶ 0:00

Cloudflare's launch post on 1 October says "the market is getting increasingly saturated with decision models". Then it releases two more, Clef and Clef-flash. Their weights are open, and they take the same requests as Jev, TypeSafe's decision model.

A decision model takes a message and a fixed list of answers, like the teams that could handle a request, and returns a probability for each answer instead of writing text. Cloudflare's post compares its two models with Jev and three others in a table of ten tests, and Clef is ahead of Jev on eight of them. The fuller table on Clef's Hugging Face card has 41 tests, and Clef is ahead on 26. Jev is ahead on the other 15, among them several general knowledge and reasoning tests (MMLU-Pro, BBH, GPQA Diamond).

A second table in the post covers four business workflows from TypeSafe's own evaluation suite, and Clef is ahead of Jev on three of them. We didn't run those.

The test we checked is in both the post's main table and the card's. BANKING77 is a public set of messages that customers sent to a bank, each filed under one of 77 reasons, like a card that hasn't arrived or a declined transfer. Its test split has 3,080 messages. On it, Cloudflare's table gives Clef 94.20 and Jev 79.74, a lead of 14.5 points. The score is macro-F1, the average over the reasons of how well the model handles each one, counting both the messages it misses and the ones it files there wrongly, and on these messages it comes out within a point of the share each model gets right. If you sort support tickets, a gap that size would make you switch. So is Clef really that far ahead, and is it worth what it costs?

The basics▶ 1:32

Cloudflare released Clef and Clef-flash on 1 October. It built them on open Qwen models, Qwen3.8-27B for Clef and Qwen3.5-9B for Clef-flash, kept those weights frozen, and trained small adapters inside them and a scoring head on top, with what its post calls "our own internal synthetic datasets". The weights are on Hugging Face under the Apache 2.0 licence, so you can run them yourself, and Cloudflare hosts both on Workers AI.

You pay for what you send in, and the answers come back free on all three. Per token, Clef costs 5.7× what Jev does and Clef-flash 2.1×. Both Cloudflare models can read images, which Jev can't. We didn't test that.

Their test, run ourselves▶ 2:01

Cloudflare's table comes from the Decision Index, a community benchmark that says it is "unofficial and community-maintained; not affiliated with TypeSafe AI". BANKING77 is one of its tests. Cloudflare ran Clef and Clef-flash on edition 0.2.1 itself and labels those scores self-reported. Its number for Jev is the benchmark's own run. So the Clef scores and the Jev score in that table come from different runs.

We sent all 3,080 messages to Clef, Clef-flash and Jev, each request exactly as the Decision Index sends it: no context, the instruction "Classify the banking intent of this user request:" with the message, and the 77 reasons as the options. We sent the same requests to Kev-27B and Kev-9B, which come up later.

Clef scored 94.20, the same as Cloudflare's table. Clef-flash scored 90.85 against the table's 90.93, and Jev 80.00 against 79.74. All three sit inside our 95% interval. Kev-9B's 84.83 in that table is an older checkpoint of it, so it doesn't compare with ours. So the table holds: on this test Clef is 14.2 points ahead of Jev in macro-F1, 13.5 points in the share right.

How we ran it: we fixed the procedure before sending a single test message, covering which messages, which request for each model, how answers are scored, how each model's cut-off for sending a message to a person is set, and how costs are counted. Clef, Clef-flash and Kev each got a short addendum on 2 October, written before their test passes, that changes only the model name and the address the request goes to. Kev's servers had answered a timing probe of the first forty test messages just before its addendum was written; nothing was set from it, and its answers weren't scored.

Their request is the Decision Index's own, copied byte for byte from its code. We sent it six at a time from a laptop in Berlin, so we don't use its times for speed. The one place they come in is Kev's cost projection below, which says so.

Our request carries the message, the five most similar messages from BANKING77's training split with their reasons, and 78 options: the 77 reasons plus "none of these", which a real queue needs and which is always wrong on this test. One fixed embedding model finds the examples in a pool of 9,405 labelled training messages, with test duplicates and our held-out messages removed, so no test message ever appears as its own example. Every model gets the same examples. We sent it to Jev, Clef and Clef-flash on the afternoon of 2 October, one request at a time from a Mac mini on a home connection in Berlin, Jev first. Kev-27B and Kev-9B ran the same evening on H100s we rented on Modal in Europe.

The cut-offs for sending a message to a person were set on 573 training messages held out from the example pool, before the test, aiming at 2% wrong answers among the messages a model routes on its own. Costs are each provider's list price times the input tokens it counted on every call, and none of the three hosted models charges for output. For Cloudflare, the neurons it reported on every call add up to the same dollars. Kev's cost is Modal's bill for our GPU time.

Five examples▶ 2:52

The Decision Index's request gives the model the customer's message and the list of reasons, and nothing else. If you sort support tickets for real, you also have old tickets that someone has already sorted. You can look up the five most like the new one and put them in the request with their right answers, which is called few-shot prompting. That is our request.

Take one test message: "Can I be given a new passcode?", filed under forgotten passcode. On their request, Jev picks change PIN at 0.63, and Clef picks forgotten passcode at 0.90. The five most similar sorted messages ("Need a new passcode.", "Where can I get a new passcode?" and three more) were all filed under forgotten passcode. With them in the request, Jev picks forgotten passcode at 1.00. Jev rounds its probabilities to two decimals.

Across the test, 437 of Jev's answers went from wrong to right and 30 the other way. To check that the examples and not our different wording carry that, an earlier run of our request with the examples taken out had Jev at 78.3%, a little under what it got on the Decision Index's request.

With five examples Jev got 93.9% right, Clef 95.1% and Clef-flash 95.2%. Clef barely needed the examples. Its lead in the share right, 13.5 points without them, is down to 1.1 points. That point isn't chance. On the same messages Clef got 54 right that Jev got wrong, and Jev 19 that Clef got wrong, and the 95% interval on the difference runs from +0.62 to +1.69 points.

Here is one each way. "How do I contact customer support about a transfer?" is filed under declined transfer. Clef gets it at 0.27 and Jev picks none of these. "Any fee for topping up?" is filed under the fee for topping up by card. Jev gets it at 0.44, and Clef picks the fee for topping up by bank transfer. At the cut-offs in "Who gets a person", all four of those answers would have gone to a person. Clef and Jev split 73 messages this way, one right and the other wrong. On 47 of them both answers were below their cut-offs, so if a person checks the unsure ones, they'd have seen most of these messages whichever model you used.

The bill▶ 5:38

In our run with five examples, sorting a million messages would cost $77.53 on Jev, $179.91 on Clef-flash and $479.75 on Clef. Most of the gap is the price per token. The rest is that Cloudflare's models count 8% more input tokens than Jev for the same request, 1,999 against 1,846.

So with examples, Clef beats Jev by about one message in a hundred for 6.2× the price, and Clef-flash gets the same point for 2.3× the price. Without examples the requests are shorter: per thousand messages, their request cost $0.448 on Clef, $0.168 on Clef-flash and $0.072 on Jev.

Speed▶ 6:17

Cloudflare's table also says its models answer faster than Jev. It gives median times of 209.3 ms for Clef, 38.8 ms for Clef-flash and 524.1 ms for Jev, which makes Clef 2.5× as fast and Clef-flash 13.5× by our arithmetic. Those times aren't measured the same way. The Decision Index says its Jev time is a "network round-trip from our lab" and not comparable with times measured on a model running on its own hardware. Cloudflare's leaderboard marks Clef's scores and latency as self-reported, with the latency "not measured on the board's hardware". Its post counts on its network for the rest: "our Clef models are hosted on Workers AI", "leading to low network latency".

So we timed every call in our run with five examples, from Berlin. Our requests entered Cloudflare's network at its Berlin edge. From somewhere else the order could change.

At the median, Clef-flash answered in 196 ms, Jev in 235 ms and Clef in 562 ms. Clef-flash had the widest spread, with the slowest calls of any model at the 99th percentile: 139 of its 159 calls over a second came in the first quarter of its pass. We ran it once and didn't look into why. For scale, a bare request with no model behind it took 171 ms to TypeSafe's host from the same machine at the start of the run.

Who gets a person▶ 7:23

In a real queue you'd let the model route the messages it's sure about and send the rest to a person. For each model we set a cut-off on the probability of its answer, using held-out training messages and aiming at 2% wrong among the messages it routes on its own. Then we ran the test with examples.

Per thousand messages, Clef sent 68 to a person, Jev 156 and Clef-flash 183. Among the messages each routed on its own, all three came out between two and two and a half percent wrong: Clef 2.40%, Jev 2.04% and Clef-flash 2.15%.

Per thousand messages, Clef costs $0.40 more than Jev and sends 88.3 fewer messages to a person, so each check it saves costs $0.0046 in API fees. At $15 an hour, that buys 1.1 s of someone's time. Unless a person can check a message in about a second, Clef is the cheaper one once you add the checking to the API bill.

Clef routes more on its own, so it also lets more wrong answers through: 22.4 per thousand against Jev's 17.2. The extra messages it routes are harder. Of the 318 that Clef routed and Jev sent to a person, Clef got 31 wrong.

How far can you take each model's probability at face value? Clef's come close to how often it turns out right (calibration error 0.030), and it is the best of the three at ranking its right answers above its wrong ones (AUROC 0.90, against 0.82 for Clef-flash). Jev's are close on average but too high whenever it is less than sure: its 481 answers below its cut-off of 0.99 averaged 0.84 and were right 72% of the time. Clef-flash's are too low. On the 736 answers it gave between 0.7 and 0.8 it was right 95% of the time. Check those two on your own data before you take their numbers as they come.

Kev 1.0▶ 9:03

Kev, from the developer Jared Palmer, is one more open-weight model that takes Jev's requests. Kev 1.0 came out on 1 October, the same day as Clef, and gathers models he had already published; its release notes say "Nothing in it is newly trained." Kev-27B is built on Qwen3.8-27B, the same base as Clef, and Kev-9B on Qwen3.5-9B-Base. Its model cards list BANKING77 among their training data, and its code trains on the training split and leaves the test split out. We ran both on H100s we rented on Modal in Europe, the same two ways as the others.

With five examples, Kev-27B got 93.8% right, level with Jev (−0.10 points, interval −0.55 to +0.36) and behind Clef (−1.23 points, interval −1.75 to −0.71). Kev-9B got 91.3%, mostly because it answered none of these 110 times on messages that have a real reason; with that option set aside its top real reason was right 93.6% of the time. On the Decision Index's request Kev-27B scored 86.41, between Jev and Clef. Two teams started from the same base model, and Cloudflare's version does far better when it sees only the message.

Kev-27B's probabilities matched how often it was right more closely than any other model's here (calibration error 0.014), on a dataset whose training split it learned from. At the same aim it sent 90 messages per thousand to a person.

We paid for Kev by the hour of GPU, not per token, so what a message costs depends on how many you send the card at once. Sent one at a time, as we did, it came to $0.7295 per thousand messages, 9.4× Jev's price, before start-up and idle time, which were 35% of Kev-27B's bill. Modal's bill works out at $4.92 per container-hour. From how fast Kev got through the Decision Index pass, sent six at a time from the laptop, six at a time would come to roughly $0.157 per thousand. That pass used shorter requests than our test, 853 input tokens a call against 975 by Kev's count, and we didn't run our test that way, so read it as a rough guide. From Berlin its median answer took 529 ms, of which the model itself took 110 ms.

Kev's own release notes put Kev-27B slightly behind Jev on datasets no Kev trained on: 52.3 against 54.0 on the release's index for datasets held out from training (breadth).

What this doesn't settle▶ 10:40

Cloudflare says it trained Clef on top of Qwen with its own synthetic data. Neither its post nor the model cards say whether BANKING77, or anything made from it, was in any of the training. Kev's cards list it. TypeSafe's models page says Jev isn't trained on customer requests and doesn't list what it was trained on.

We ran a check on Clef and Clef-flash that compares 200 training messages they might have seen with reworded copies of them. Both did better on the originals, but so does a vote of the five nearest labelled examples, which can't remember anything (+15.0 points), and both gaps were smaller than that. So the check can't tell memory from the cost of rewording. That matters most for the result without examples, where Clef's lead is big.

We tested one task, in English. In the Hugging Face table Jev is ahead of Clef on 15 of the 41 tests, so other tasks may come out differently. We didn't test images. On a second pass over the same messages, Clef and Clef-flash gave the same answer to 200 of 200. And this is one day of calls, from Berlin.

Which to use▶ 11:33

If you have no sorted messages to show the model, use Clef. It was 14.2 points ahead of Jev on the Decision Index's request, and Clef-flash 10.8 points, for 38% of Clef's price.

If you do have sorted messages, Clef, Clef-flash and Jev come within about a point of each other, and the other differences decide it.

For the smallest bill, use Jev, at about a sixth of Clef's price per message.

If a person checks the answers the model isn't sure about, Clef sends them fewer than half as many messages as Jev. Unless a check costs you less than $0.0046, that makes Clef the cheaper one overall, with a few more wrong answers getting through.

If no person checks the answers, Clef-flash got as many right as Clef in our run, for 38% of Clef's price and 2.3× Jev's.

For speed from Berlin, Clef-flash and Jev were close at the median, and Clef was the slowest.

If you want to run one on your own hardware, Clef, Clef-flash and Kev are all open, and Clef and Clef-flash were the most accurate of them here. Kev-27B matched Jev with examples, but it learned from this dataset's training split, and its own release notes put it slightly behind Jev on data no Kev trained on. What it costs depends on how you host it.