Published
Updated
A v3 judge paper puts a frozen Jev cascade 0.9 points above GPT-6 on 1,610 pairs
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman revised arXiv:2609.26550 to v3 on September 29. On 4,850 public judgments JEV is eighth of fifteen LLM judges, 1.7 points behind GPT-6, at $0.044 per 1,000 judgments. A frozen cascade is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs, at 41% of its fee. We did not rerun it.
Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman revised arXiv:2609.26550 to v3 on September 29, 2026, at 15:32 UTC. v1 was submitted September 22. The HTML license is CC BY 4.0. Cristian Leo posted a pointer on September 28 that quotes an RM-Bench hard cell of 74.8% against a normal cell of 84.0%, AUROC 0.770 against 0.902, and error rates in two high-confidence bands. We did not find that pair in the v3 sections we opened. The numbers below are the v3 text. The post has no photo we saved.
Where the verdict can be read off the text, the paper says JEV stays within three points of GPT-6, at 0.36% of its fee and a median of 0.15 seconds, and falls behind where the verdict has to be derived. It names math, code, and logic for that second case. A frozen cascade is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs, at 41% of its fee. A pre-specified live test on two new workloads matches GPT-6 exactly.
On 4,850 pooled public judgments, JEV is eighth of fifteen LLM judges and 1.7 points behind GPT-6, at $0.044 per 1,000 judgments. The paper calls that 277 times cheaper than GPT-6. The median is 0.15 seconds. The introduction puts GPT-6 at low effort at about $12 per 1,000 judgments. At confidence of at least 0.9, the paper prints 94.7% against 94.4% on 3,744 items. Style-adversarial AUROC of the confidence score is 0.76 against 0.84 on style-matched pairs.
The cascade escalates 31% of the 1,610 pairs. On RewardBench it escalates 25% and spends 27% of GPT-6’s fee. On JudgeBench, JEV alone trails by 12 points. The cascade escalates 65% and closes 94% of that gap. A live run on 510 of these pairs has a median of 0.27 seconds, against 2.10 seconds for GPT-6 alone. Among eight inexpensive or open first stages, the paper says JEV is the only one whose cascade stays within two points of GPT-6 on both benchmarks and saves at least 30% of the fee on each.
The prospective live test uses 570 held-out pairs from PPE and from JudgeBench’s Claude split. JEV trails GPT-6 by 14 points. The cascade matches GPT-6 exactly, escalates 74%, and saves a quarter of the fee. The thresholds come from 100 local labels per workload.
Table 1 prints base accuracy for JEV 1.13 as RewardBench 92.5 on 1,500 items, JudgeBench 78.6 on 350, HaluEval 87.3 on 3,000, and final answer 94.0 on 150, with 5,172 of 5,172 judgments valid, a fee of $0.044 per 1,000, and a median of 0.15 seconds. GPT-6 Astra on the same columns is 92.5, 93.1, 88.4, and 96.7, with 5,171 of 5,172 valid, a fee of $12.182, and a median of 1.89 seconds. GPT-5.6 Sol is 1.1 points higher than GPT-6 on the 4,850 pooled public judgments, at half the fee. JEV is 2.8 points behind GPT-5.6. p95 is 0.24 seconds for JEV and 3.86 seconds for GPT-6.
The RewardBench paired difference against GPT-6 is 0.0, with an interval from -1.4 to 1.4. HaluEval is -1.1, interval -1.9 to -0.3. JudgeBench is -14.6, interval -18.9 to -10.3. On a 990-item human adjudication, readers side with GPT-6 on 57 of 69 disputed JudgeBench items and with JEV on one, with 11 indecisive. The human-adjudicated difference is -16.0. On RewardBench disagreements, GPT-6 takes 17 of 29, JEV takes 5, and 7 are indecisive. The timing panel is 120 judgments per hosted judge. The paper counts sixteen other judges, fourteen of them generative and two of them reward models. The base set of 5,172 is a 642-item pilot plus 4,530 held-out items. The 1,610 preference pairs are 1,340 from RewardBench and 270 from JudgeBench.
The abstract page says an earlier claim, “99% of GPT performance at 57% of the cost,” stays in the appendix. We did not open that appendix, and we did not open a code repository for the paper. We did not rerun the tables. This paper is not the LangChain weather-agent repository danielgshea/jev-as-a-judge.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The HTML we read is arXiv:2609.26550v3. Authors are Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. v1 was submitted 2026-09-22. v3 was revised 2026-09-29T15:32:52Z. The license on the page is CC BY 4.0. Cristian Leo's post on September 28, 21:45 UTC, points at the paper and quotes an RM-Bench hard cell of 74.8% against a normal cell of 84.0%, a gap of 9.2 points, AUROC 0.770 against 0.902, and error rates in the bands 0.90 to 0.95 and 0.95 to 0.99. We did not find that RM-Bench hard/normal pair in the v3 sections we opened. The figures below are the v3 text.
Where the verdict can be read off the text, JEV is within three points of GPT-6, at 0.36% of its fee and a 0.15 second median, and behind where the verdict must be derived, which the paper names as math, code, and logic. A frozen cascade is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs, at 41% of its fee. A pre-specified live test on two new workloads matches GPT-6 exactly.
Pooled, 4,850 public judgments: JEV is eighth of fifteen LLM judges, 1.7 points behind GPT-6, at $0.044 per 1,000 judgments, which the paper calls 277 times cheaper than GPT-6, median 0.15 seconds. The introduction puts GPT-6 at low effort at about $12 per 1,000 judgments. At confidence at least 0.9, the paper prints 94.7% against 94.4% on 3,744 items. Style-adversarial AUROC of confidence is 0.76 against 0.84 on style-matched pairs.
The cascade escalates 31% of the 1,610 pairs. On RewardBench it escalates 25% at 27% of GPT-6's fee. On JudgeBench, JEV alone trails by 12 points, the cascade escalates 65%, and it closes 94% of the gap. A live run on 510 of these pairs prints a median of 0.27 seconds against 2.10 seconds for GPT-6 alone. Among eight inexpensive or open first stages, the paper says JEV is the only one whose cascade stays within two points of GPT-6 on both benchmarks while saving at least 30% of the fee on each.
The prospective live test is 570 held-out pairs, PPE and JudgeBench's Claude split. JEV trails GPT-6 by 14 points. The cascade matches GPT-6 exactly, escalates 74%, and saves a quarter of the fee. Thresholds come from 100 local labels per workload.
Table 1 base accuracy. JEV 1.13: RewardBench 92.5 (n=1500), JudgeBench 78.6 (350), HaluEval 87.3 (3000), final answer 94.0 (150), valid 5172/5172, fee $0.044 per 1,000, median 0.15 seconds. GPT-6 Astra: 92.5, 93.1, 88.4, 96.7, valid 5171/5172, fee $12.182, median 1.89 seconds. GPT-5.6 Sol is 1.1 points higher than GPT-6 on the 4,850 pooled public judgments, at half the fee. JEV is 2.8 behind GPT-5.6. p95 is 0.24 seconds for JEV and 3.86 seconds for GPT-6.
RewardBench's paired difference against GPT-6 is 0.0, interval -1.4 to 1.4. HaluEval is -1.1, interval -1.9 to -0.3. JudgeBench is -14.6, interval -18.9 to -10.3. Human adjudication of a 990-item subset sides with GPT-6 on 57 of 69 disputed JudgeBench items and with JEV on one, with 11 indecisive. The human-adjudicated difference is -16.0. On RewardBench adjudication, GPT-6 takes 17 of 29 disagreements, JEV 5, and 7 are indecisive. The timing panel is 120 judgments per hosted judge. The paper counts sixteen other judges, fourteen generative and two reward models. The base set of 5,172 is a 642-item pilot plus 4,530 held-out items. The 1,610 preference pairs are 1,340 RewardBench plus 270 JudgeBench.
An appendix comment on the abstract page says a previous "99% of GPT performance at 57% of the cost" is retained in the appendix. We did not open that appendix. We did not open a GitHub repository for this paper. We did not rerun the tables.
Compare
danielgshea/jev-as-a-judge is a LangChain weather-agent eval on frozen traces. This paper is a different project. Decision-hijack, arXiv 2609.28613, is an attack paper on 510 InjecAgent cases. JevAdvBench, arXiv 2609.31142, is a third paper, on wording and an appended opinion.
The $0.044 per 1,000 judgments is the paper's fee for JEV 1.13 on this protocol. Jev's published list price on this desk is $0.042 per million input tokens, output free, which is a different unit. The 0.15 second median is the paper's, not JevBench's 0.62 second p50.
Terms
- 1.7 points
- How far JEV sits behind GPT-6 on the paper's pooled 4,850 public judgments, eighth of fifteen LLM judges. The fee on that line is $0.044 per 1,000 judgments.
- 0.9 points
- The frozen cascade's accuracy margin over GPT-6 on 1,610 held-out pairs, at 41% of GPT-6's fee. The cascade escalates 31% of those pairs.
- 5172/5172
- Valid JEV 1.13 judgments in Table 1. GPT-6 Astra is 5171/5172. The base set is a 642-item pilot plus 4,530 held-out items.