Jev keeps the same direction on 28,074 classical-Korean sentences

vase.park posted on September 23 that a Jev pass over the same classical-Korean sentences came out close to the earlier chat-model panel, and from the numbers looked roughly between GPT and Claude. The link is the results note in hw725/dansa-research. The repository studies sentence-ending patterns in Korean translations of classical Chinese. The root listing has no LICENSE file. The translations themselves are not in the public tree. The note says the panel statistics can be recomputed from anonymized keys.

Three chat models had already marked every item: GPT-5-mini, Gemini 2.5 Flash, and Claude Sonnet 4.6. They received twenty sentences in one prompt and wrote answers a parser turned into O or X. Jev (jev-latest) returned one probability per sentence. The run sent one sentence at a time, so the neighboring nineteen sentences were not in the state. A probability of at least 0.5 counts as O. The input is the translation only. The classical original and the phrase-reading marks stay out.

The set is 28,074 items. Section 1, 로다 against 라, is 2,606 target and 2,606 control. Section 2, 니라 against 라, is 11,135 each. Section 3, 하나니라 against 라, is 296 each. Jev’s O rates, target then control, are 43.6% and 5.3% (+38.3 points), 34.4% and 23.0% (+11.3), and 86.8% and 50.0% (+36.8). GPT-5-mini’s gaps are +46.5, +3.8, and +36.8. Gemini 2.5 Flash’s are +39.3, +1.0, and +22.0. Claude Sonnet 4.6’s are +47.0, +18.5, and +31.8. On section 2, Jev’s O rate is 34.4%, under the chat models’ 52% to 77%. The note calls Jev the most conservative of the four judges.

Adding Jev as a fourth vote changes the unanimous counts and leaves the direction in place. Book-stratified Mantel-Haenszel odds ratios move from 7.67, 2.27, and 2.04 to 8.64, 2.17, and 1.78. The note says every 95% interval stays above 1. Fleiss’s kappa moves from 0.498, 0.299, and 0.645 to 0.463, 0.237, and 0.608. Unanimous O-rate gaps move from 34.8, 18.0, and 34.5 points to 24.9, 11.4, and 35.5. With four judges, unanimous means four votes, so that gap is not the same quantity as the three-model gap. Cohen’s kappa of Jev with GPT, Gemini, and Claude is 0.568 / 0.323 / 0.442 on section 1, 0.176 / 0.163 / 0.363 on section 2, and 0.645 / 0.525 / 0.555 on section 3.

The published canonical numbers in final_stats_v3.1_cleaned_balanced.json were not replaced. The Jev panel is a side check. The note says there is no gold label, English is the model’s primary training language, and Korean accuracy is not guaranteed by the vendor docs. What the table shows is that each section still has a higher O rate on the target ending.

342 probabilities land on 0.5 exactly: 49, 291, and 2. They were counted as O. A rule of strictly above 0.5 moves Jev’s own gaps to 36.7, 11.5, and 37.5 points. A morning trial on section 3 only, twenty sentences per state, separated unanimous O from unanimous X at AUC 0.954. An inverted question was yes on both framings for 0.5% of items, and the two probabilities correlated at -0.91. That trial was replaced by the one-sentence run.

A Solar Mini 4 wrapper, hunkim/solar-mini4-jev, was tried on section 3 under the same one-sentence rule. Target O was 99.3% and control O was 95.3%, a gap of 4.1 points, with kappa 0.07 to 0.13 against the three chat models. 576 of 592 items received 0.95. The only other probability in the file is 0.05. Sections 1 and 2 were not run. The note says the wrapper prompt includes few-shot values near 0.05 and 0.95, and that _heuristic_overrides can replace a probability from benchmark keywords, so a bench line of Solar 98.4% against Jev 94.2% does not carry over.

The run log records a Cloudflare 403, error 1010, from 08:48 to 11:34 UTC on September 23, then 15 HTTP 520 responses that were retried. Jev usage is 28,089 calls, about 12.79 million input tokens, about $0.54. The Solar section 3 spend is listed at about $0.08. We did not repeat the calls.