AI judging AI: what JEV can tell you about a model’s answer

One answers, the other checks

You have probably heard of LLM-as-a-Judge. So have we. If you have not: it means asking one model to judge another model’s answers. We give it a question, an answer and rules for checking it. A little like marking homework, except the teacher can make mistakes too.

Imagine a store assistant. A customer asks where their parcel is, and the assistant beautifully explains the returns policy. It all sounds sensible, but the parcel is still missing. A judge is meant to notice exactly that: an answer can be well written and completely miss the question.

This is where JEV comes in, spelled Jev by its maker. It is a decision model from TypeSafe. Instead of writing a review of the answer, it returns a result in a defined format: a category, a rating on a described scale or the probability that a condition is met. It does not add an explanation. That is useful if we want to automatically pass an answer or send it for review. If we want to know what to improve in the wording, JEV alone will not explain it.

Comet compared two judges. Here is the interesting part

Our starting point was Comet’s article from September 29, 2026. The authors compared Jev with GPT-4o-mini on 1,000 conversation turns from 578 threads with their production assistant Ollie. The question was simple: does the answer address what the user asked? The models agreed in 897 cases, or 89.7%.

It is easy to read that as “almost 90% correct judgments.” But the study did not establish that. The authors say they do not know which judge was right. We know how often they said the same thing. Both could also have passed the returns-policy answer when someone asked about a parcel.

Comet also reported cost and time: judging the whole set cost $0.0285 with Jev and $0.0993 with GPT-4o-mini. Median latency was 253 and 958 ms respectively, while p90 — the boundary containing 90% of measurements — was 352 and 1,202 ms. These are results for their task and environment. Your application needs its own measurements.

Before asking the model, agree on what “good” means

“Judge whether this answer is good” leaves plenty of room for interpretation. Start with separate questions: does it address the request? Are its facts in the supplied material? Does it tell the user what to do next, if the task requires that? Then we can see where the problem lies. An assistant can answer the right question while inventing a delivery date.

One more thing: sometimes “I don’t know” is a good answer. If the assistant lacks an order number, asking for it makes more sense than guessing the parcel’s status. Write that rule before judging. Otherwise, we will argue with the model over a rule we never established ourselves.

If you want to connect JEV to your project

Through OpenRouter, send a POST request to https://openrouter.ai/api/alpha/decisions. Choose typesafe/jev-1.13, put the material to evaluate in state and the questions in questions. This is a separate format: an ordinary chat call with a messages array is not enough. Keep the OpenRouter key on the server.

For a yes/no condition, choose the noul type. The result appears in answers under your question’s name and its noul field. You get a number from 0 to 1 without a separate confidence field. Then decide which scores pass an answer and which send it to a person. A score of 0.9 alone does not guarantee correctness; check the threshold on your own examples with established judgments. Record the model version, cost and call duration too, so there is something to revisit.

All right, let’s judge a few answers ourselves

JEV has limitations too. TypeSafe describes literal reading of instructions, difficulties with counting and comparing dates, and susceptibility to text trying to influence the judgment. If we need a word count, let’s calculate it in code. Leave the model to judge meaning, with a short source and a clear rule.

We prepared a small lab: “Spot the mistake: you, JEV and GPT-6 Luna.” There are 12 examples. First, decide for yourself whether an answer should pass. Then reveal our reference judgment and the recorded results from both models. A slider changes the threshold so you can see which mistakes get through and which correct answers are stopped.

The examples and reference judgments were created with AI assistance and concern fictional situations. This is an exercise in understanding the mechanism, not an independently human-rated test or model ranking. Open it in “Make it yourself,” and look closely at the answer that turns free admission into no booking required.