Choosing a decision model for an agent harness: Clef vs Kev, Laya and Open-Jev

Cloudflare’s Clef-Flash made the best decisions of the four open decision models we tested for the Rameness agent harness. It got all 54 standard cases right and 89% of the reworded ones, in 0.15 s each on an RTX 5070 Ti. Kev-4B is the best choice without a GPU, at 93% and 83%, though it is ten times slower and less sure of its answers.

What a decision model does

A general-purpose LLM is built to generate language: give it a situation and it can explain, qualify and suggest. A decision model has a narrower job. Give it the same situation plus a fixed set of allowed answers, and it returns one choice—or true/false—with a probability for every option. That makes the result easy for software to log, compare and act on.

The model underneath can still understand language. The difference is the job it is given and the constrained output the harness receives.

Why the LLM doesn’t make the decisions

In Rameness, every multiple-choice fork in the agent loop goes to a small typed decision model, never to the LLM doing the work. That layer is called JEV. It decides things like:

• Route: answer directly, run a tool, or start an agent.
• Effort: how much reasoning a task needs, and how hard to think before the next step.
• Significance: is this decision routine, or should a person make it (deploys, deletions, emails to clients)?
• Recovery: after a failure, a stall or a loop, whether to retry, wait, reorient or escalate.
• Intake and categories: how the manager splits incoming work, and which of 36 task categories it belongs to.

A typed model reads a short state and returns a probability for each option in one forward pass. No text is generated, so it can’t ramble, can’t produce an answer outside the options, and is fast enough to call on every turn. We tried letting the LLM decide early on; it was slower and less consistent. The probabilities also tell us how sure each decision was, and every decision is logged.

The contenders

Four open decision models that speak the Jev “System One” API, plus a baseline with no model at all.

The lexical baseline is Rameness’s last-resort fallback when no model is reachable. It shows how much of the test can be solved by matching words alone. TypeSafe’s hosted Jev was not tested: it is a paid service, and we only compared models you can run yourself.

How we tested

Each model answered the same 72 labelled decisions, taken from the questions Rameness really asks.

• 54 standard cases across 10 decision kinds: scope, route, effort, significance, intake, failure, stall, question, loop and task category. Each case has the harness’s real question and options, a short state, and the answer a careful person would give. The cases are clear-cut on purpose: a model that misses them will miss the ambiguous ones too.
• 18 paraphrased cases: the same situations reworded so they don’t share words with the options. These separate understanding from keyword matching.
• Two wordings of the options: Rameness can describe each option as a few keywords or as a plain sentence. Every model was asked both ways.
• Batching: one request carrying 1, 4, 8 and 16 yes/no questions, which is how Rameness asks “which of these apply?”.

Hardware: an AMD Ryzen 7 7800X3D (8 cores) with 30 GB of RAM for the CPU models, and an RTX 5070 Ti for Clef-Flash. Clef-Flash is built for a GPU, so the comparison is each model where you would actually run it. The CPU models were started fresh one at a time; Clef was measured on its running server. Each model got the same state budget Rameness gives it in production.

Results

Rameness JEV benchmark, October 15, 2026 (Open-Jev: September 25, 2026) · 72 labelled decisions.

The reworded cases are the real test. The lexical scorer looks strong on standard phrasing because its cues were written for it, then falls to 61% once the words change. Kev holds up well but gives the right answer less confidence: it put on average 67% of its probability on the correct option, against Clef’s 88%.

Where each model misses

Clef-Flash’s only misses are two near-synonyms among 36 task categories. Kev’s misses are the ones that matter in a harness: safety and recovery.

Accuracy per decision kind, standard and reworded cases together, each model with its production wording:

What the misses were:

• Clef-Flash: “the page is sluggish when lots of people are on it” became performance testing instead of performance, and “make sure screen-reader users can use the signup page” became accessibility instead of accessibility testing.
• Kev-4B: it called “send the invoice email to the client?” routine, when an email to a client should go to a person. After 40 failed turns it forked the agent instead of moving to a stronger model. On a tool call failing the same way six times, it stopped instead of reorienting.
• Laya: 21 misses, including rating “merge into main and deploy to production” as routine and giving “list the files in src” high effort.

Open-Jev-2B isn’t in this table: its run used an earlier version of the set, so its per-kind numbers don’t line up.

Speed and footprint

On a GPU, Clef-Flash decides in 0.15 s, ten times faster than Kev on a CPU. The catch is that it needs about 12 GB of GPU memory.

Batching matters because Rameness asks “which of these apply?” as one yes/no question per option. Clef answers 16 at once in barely more time than one. Kev’s long tail comes from the number of options: about 1.5 s for a 2-to-5-way choice, but 7.2 s to pick among the 36 task categories. That adds up over hundreds of decisions in a run.

Keywords or sentences

How you word the options matters, and not the same way for every model. Clef-Flash is better with plain sentences; Kev is slightly better with keywords.

With keywords, Clef missed two intake decisions: “migrate the whole monolith to microservices” (should be split into parts, not researched) and “make it better” (should get a clarifying question). Described as sentences, both came out right. So Rameness writes each option as a sentence for Clef and as keywords for the others. Laya was measured with keywords only.

What we picked, and the caveats

Rameness now picks Clef-Flash first whenever it is running, then Kev (the CPU option), then Laya (the lightweight one). If no model is reachable, the lexical scorer keeps the harness running, with a warning.

What this test doesn’t tell you:

• The set is small and clear-cut: 72 cases, 4 to 11 per decision kind. One miss moves a kind by 10 to 25 points.
• The hardware isn’t equal: Clef ran on a GPU and the others on a CPU, which is how we’d deploy each. Kev and Laya on a GPU would be faster, not more accurate.
• Open-Jev is under-tested: only its 2B model was measured, in an earlier run on an earlier set. Its 9B and 27B releases are the fairer match for Clef and weren’t tested.
• No hosted models: TypeSafe’s Jev was left out.

To reproduce it, start Clef with rameness jev up clef and run:

python -m bench.jevmodels --models clef,kev,laya,openjev,lexical --styles both

The cases are in rameness/jevbench.py in the Rameness repository, and rameness jev bench checks whichever model you have configured.