Would you let this answer through?
“The ticket costs 35 zł.” Short, specific, time to plan the evening. Except the event description says nothing about a price. The assistant added it. This mistake is easy to catch, but less obvious answers follow: the correct concert time with the backpack question omitted, or free admission suddenly taken to mean no booking is needed.
We gathered 12 such situations and sent them to JEV 1.13 and GPT-6 Luna. Before looking at the results, try them yourself. The exercise above shows a source, question and answer. Choose “Pass” or “Stop”; only then do the reference judgment, explanation and model results appear. The controls are in English; the case titles, sources, questions, answers and reference explanations stay in Polish to match the actual model runs.
The places and events are fictional. We prepared the examples and reference judgments with AI assistance; they are not customer cases or independent human assessments. The aim is to see how a judge works and where you might disagree. Twelve examples are not enough to declare the best model.
What we actually asked them to check
Both models had the same task: check whether the answer covers every part of the question, follows the source and adds nothing unsupported. If the source omits something, the answer can say so directly. With a missing price list, “the price was not provided” is fine. Guessing an amount is not.
Through OpenRouter, we sent only the source, question and answer — the source, question and answer fields. We kept our judgments and explanations to ourselves. The models did not receive example titles or the reference and reason fields that could have given them hints.
There is a difference between the numbers too. JEV returned Noul through Decisions API; we asked Luna for a probability number in JSON through the chat API. The criterion was shared, but the way of obtaining a score was different. We did not test calibration: whether these numbers correspond to the actual frequency of correct judgments. So we do not read 0.9 as a promise of 90% accuracy.
A tie at 0.5. Now move the slider
After the quiz, open “Set the threshold.” An answer passes when the model’s score is equal to or above the threshold. We started at 0.5. Both models then agreed with our reference judgments in all 12 cases.
Lower it to 0.2 and see what happens. Each model lets one mistake through, but a different one. JEV accepts example 10, where the assistant assumes that free participation means no booking is needed. It scored that answer 0.25. Luna gave the same score to example 4: the concert time is right, but the answer ignores the question about a large backpack.
Now go the other way: 0.95. JEV stops the correct answer about the gallery next to the café, example 12, which scored 0.94. Luna still passes all six correct answers, each scored 1.0. This shows the cost of tightening the rule: we can stop something good along with the mistakes. The slider makes that decision visible on specific texts.
Take the whole exercise with you
The ZIP contains the original Polish project: examples, results, API call records, code, instructions and an offline version of the lab. Open laboratorium.html in a browser. The quiz and slider use recorded numbers, so they need no account or key and do not send anything to the models again.
To change the examples and ask the models again, you need Python 3, an OpenRouter key and account funds. The evaluate.py script requires no additional libraries. Extract the archive and open a terminal in its folder. The first command below validates the data without a network connection. The second makes paid calls and saves a new result.
Before the second command, set OPENROUTER_API_KEY in your terminal environment. The README shows how to enter the key without displaying it on screen. Do not add it to code or files you will later share.
What if you changed just one sentence?
Take the booking example and add to the source that booking really is not required. Or complete the answer about the backpack. Then ask the models again: was that one change enough to move the score?
Edit the examples in cases.json. A new example needs a unique id and the fields title, type, source, question, answer, reason and the boolean reference judgment. Write your own decision and explanation first, then run the models. Choose a new output filename for each run: the script does not overwrite earlier results.
The results also contain call durations. These are individual measurements including the network and OpenRouter, so we are not turning them into a speed contest. If you would first like to understand the idea of AI judging AI, visit the related guide “AI judging AI.”
Sources and further reading
The sources behind this article. Instructions and example scenarios are suggestions from our editorial team.
- Cookie Analytics: recorded JEV and GPT-6 Luna results — October 3, 2026 (opens in a new tab)qkianalytics.pl
- Cookie Analytics: complete lab archive, data and code (opens in a new tab)qkianalytics.pl
- Cookie Analytics: AI judging AI — what JEV can tell you about a model’s answer (opens in a new tab)qkianalytics.pl
- OpenRouter: Decisions API — Noul response format (opens in a new tab)openrouter.ai
- TypeSafe: System One and the meaning of calibration (opens in a new tab)docs.typesafe.ai