Published
LangChain ran Jev as a judge: 500 matching pass/fail labels, and a $0.34 bill against Claude's $28
Daniel Shea and Seán Roche compared Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 on five frozen weather-agent traces. Jev matched a human pass/fail label on all 500 repeated decisions. Continuous quality-score variance was 92 to 913 times lower than the LLM judges. Five cases, one reviewer.
LangChain posted on September 19 that Daniel Shea and Seán Roche had run Jev as an agent evaluator against three LLM judges. The code, frozen traces, and oracle labels are in danielgshea/jev-as-a-judge. Sydney Runkle quoted the post.
They built a weather agent with Deep Agents and Tavily, ran five requests once (Seattle now, Austin weekend, Dublin umbrella, Tokyo extended forecast, Springfield with no unique place), and stored the full traces. After that, only the judges could vary.
Each judge scored those five traces 100 times. Two signals: quality, a 0–1 score used only to measure repeatability, and does_pass, a binary used for accuracy against a human label.
On does_pass, Jev matched the oracle on all 500 decisions. Terra 99.8%, Luna 96.4%, Claude Sonnet 4.6 80.0%. The README says this is five runs and one reviewer.
On the continuous score, Jev’s mean per-case variance was 0.0000149. Luna was 433 times higher, Terra 913 times, Claude 92 times. The authors do not claim the architecture caused that. They note a consistently wrong judge would also look stable.
Bills for the evaluator calls: Jev $0.34 total at $0.00035 a call and 0.44 s. Luna $0.39 at $0.00039 and 2.50 s. Terra $2.90. Claude $28.17 at $0.02811 a call. The per-call gap with Luna is small; the gap with Claude is the one the post leads with.
LLM judges ran through LangSmith Gateway at each provider’s defaults. Jev ran through langchain-typesafe==0.0.1a2. The experiment metadata does not name the hosted Jev version. We did not rerun the eval.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The September 19 LangChain post is the write-up. The repo is danielgshea/jev-as-a-judge. Five weather requests (Seattle, Austin, Dublin, Tokyo, Springfield) were run once through Deep Agents plus Tavily, then frozen. Each judge scored those five traces 100 times with a continuous quality score and a binary does_pass. A human reviewer labeled the five responses; that label is the oracle for binary accuracy. does_pass agreement: Jev 100.0% (500/500), GPT-5.6 Terra 99.8%, GPT-5.6 Luna 96.4%, Claude Sonnet 4.6 80.0%. Mean quality variance: Jev 0.0000149; Luna 0.00647 (433×); Terra 0.01364 (913×); Claude 0.00137 (92×). Cost and latency: Jev $0.00035 and 0.44 s ($0.34 total); Luna $0.00039 and 2.50 s ($0.39); Terra $0.00289 and 2.83 s ($2.90); Claude $0.02811 and 2.16 s ($28.17). LLM judges went through LangSmith Gateway at provider defaults; Jev through langchain-typesafe==0.0.1a2. The hosted Jev version is not in the experiment metadata. The README says this is a small corpus with one reviewer. We did not rerun it. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; this story reports the post and repo as published.
Compare
Mike Taylor's Every piece asked Jev 21 writing questions on 37 documents and did not put an LLM judge on the same 777 cells. LangChain does, on five frozen traces. Lemkin's SaaStr check is the other side of the same job: Jev was cheaper than Sonnet and let more weak answers through. Luna's per-call price here ($0.00039) is close to Jev's ($0.00035); the gap that remains is latency and variance, then Claude's $28.17 total. The 100% binary match is five labeled traces times 100 repeats, not a new public set.
Terms
- LLM-as-a-judge
- A generative model scores another system's trace. LangChain compared that pattern with Jev on five frozen weather-agent runs.
- does_pass
- The binary evaluator in jev-as-a-judge. Jev matched the human oracle on all 500 repeated decisions in this run.
- Quality-score variance
- How much a judge's continuous 0–1 quality score moved when the agent output was unchanged. LangChain reports Jev at 0.0000149 mean per-case variance.