On DeepEval's Paris weather example, jev-1.13.0 scored 0.254 when the answer invented a breeze

Sumanth posted on October 13, 2026, at 09:43 UTC, that JevEval uses Jev as the judge for custom LLM evals. The caller writes the decisions. Jev returns probabilities. DeepEval turns those answers into a score. A comment on that post links Introducing JevEval. Jeffrey Ip, who builds DeepEval at Confident AI, replied that JevEval is a step up to using Jev out of the box.

The metric is JevEval on a single-turn LLMTestCase, and ConversationalJevEval on a conversation. You list the test-case fields Jev is allowed to see, then a list of Noul, Score, and Choice questions, each with an optional weight. DeepEval sends that state and every question in one decide() call. Jev returns a probability distribution per question. The score is a weighted mean over the questions that applied:

JevEval = sum(w_i v_i) / sum(w_i)

A Noul is a yes/no proposition. Its value is P(true). A Score is ordered levels, worst first. Its value is the expected level divided by n minus 1. A Choice maps each option to a credit in [0, 1], or to None when the question should not count. If the None options hold half or more of the probability, that question drops out. Credits stay in DeepEval. The option names go to Jev.

The docs build a Tool Faithfulness metric on a weather tool that returned 18°C and sunny, while the answer also claims a light breeze and 40% humidity. Four questions: every fact appears in a tool output, weight 2; returned values are reported accurately; how grounded the answer is, four levels from Fabricated to Fully grounded; what the answer did with facts the tools never returned, with stated_it_as_fact at credit 0 and nothing_missing at None. The first question carries twice the weight because an unsupported fact is the failure the example cares about most. That weight is in the constructor.

Jeffrey introduced Jev inside DeepEval on September 22, 2026, at 16:22 UTC. The GitHub release python-v4.2.4, 16:36 UTC that day, is titled Introducing Jev in DeepEval and names version 4.2.2. Existing metrics can run hybrid, where an LLM still extracts claims and Jev decides each verdict, or system_one, where Jev judges the raw test case. G-Eval stays an LLM judge. JevEval is the custom metric with no LLM in the path, including the reason string, which DeepEval builds from the answers. On September 28, at 18:55 UTC, Jeffrey introduced JevEval in Python and TypeScript. Opened October 14, 2026, PyPI deepeval 4.2.8 is Apache-2.0, author Jeffrey Ip, wheel uploaded 2026-10-02T15:49:15Z. The README on that page names JevEval. LICENSE.md on main is Apache License 2.0, Copyright 2024 Confident AI Inc.

This desk did not install deepeval. On October 14, 2026, the Date header 01:58:11 GMT, two POSTs to https://api.typesafe.ai/v1/systemone with model jev-latest returned HTTP 200 and named jev-1.13.0. The state and the four questions follow that Tool Faithfulness example. The credits were applied here after the response. There was no processing-time header.

On the invented-breeze answer, Noul P(true) was 0.03 for “every fact appears in a tool output” and 0.63 for “returned values reported accurately.” The Score was 1.74, confidence 0.72, with 0.78 on Mostly grounded. The Choice was stated_it_as_fact at probability 1.0, confidence 1.0. Mapping those answers through the published equation, weights 2, 1, 1, 1, gives 0.254. strict_mode is 0. The least decisive Noul-style confidence is 0.26, on the 0.63. Usage was 545 input tokens and 123 output tokens. The desk clock was 478 ms.

On the clean answer, “It’s 18°C and sunny in Paris.”, Noul P(true) was 0.82 and 0.97. The Score was 2.98, confidence 0.99, with 0.99 on Fully grounded. The Choice put 0.89 on nothing_missing, so that question dropped. The three remaining questions give 0.901. strict_mode is 1. Usage was 533 input tokens and 120 output tokens. The desk clock was 478 ms. List-price arithmetic at $0.042 per million input tokens, output free, is $0.000023 and $0.000022.

The docs walkthrough prints 0.30, 0.90, 0.567, and 0.237 on those four questions and a mean of 0.461. It calls that distribution illustrative. This desk’s invented-breeze call is a different set of probabilities on the same questions.

strict_mode=True replaces the mean with 1 only when every applicable question is answered in its best way from Jev’s probabilities: Noul P(true) at least 0.5, Score’s most probable level the top level, Choice’s most probable applicable option at credit 1.0. Omit evaluation_params and single-turn JevEval sends a compacted agent trace instead of test-case fields. ConversationalJevEval uses the same three question types on a turns list. G-Eval still asks an LLM to draft steps from a one-line criteria. JevEval has no model argument for a generative judge.

This page is DeepEval’s metric plus two hosted calls. LangChain’s five-trace eval and the September 29 cascade paper stay on their own pages. The JevEval pages we opened do not print agreement with a human rater. The September 22 blog says DeepEval will soon release benchmark results for Jev across metrics. Opened October 14, that comparison was not on the JevEval docs page.