Published
Updated
A Japanese table has Jev at 96.0% on one set and 56.0% on JamC-QA
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
t.toda posted a three-column table. Jev is 96.0% on JCommonsenseQA, 90.2% on JMMLU, and 56.0% on JamC-QA, mean 80.8%. GPT-5.6 Luna is 97.2%, 82.5%, and 76.5%, mean 85.4%. Bonsai 2, read as a one-token logit, averages 67.2%. The image prints no item counts and no prompt.
t.toda posted a Japanese benchmark table on September 22. The note says Jev had been described as not very capable, so the sets were worth running, and that general knowledge looks ordinarily strong. JamC-QA is the column that did not. The post’s reading is that academic questions can go well while questions about fine-grained knowledge specific to Japan go worse. A one-token logit from Bonsai 2, which the author had been trying, sits on the same table and does not look like a stand-in.
The image prints four columns: JCommonsenseQA, JMMLU, JamC-QA, and the average of those three.
| Model | JCommonsenseQA | JMMLU | JamC-QA | Mean |
|---|---|---|---|---|
| Jev | 96.0% | 90.2% | 56.0% | 80.8% |
| Bonsai 2 (1-token logit) | 93.8% | 66.8% | 41.2% | 67.2% |
| GPT-5.6 Luna | 97.2% | 82.5% | 76.5% | 85.4% |
Jev leads Luna on JMMLU, 90.2% to 82.5%, and trails on JamC-QA, 56.0% to 76.5%. The three-set mean is 80.8% against 85.4%. Bonsai 2 is close on JCommonsenseQA and well behind on the other two. The mean column matches an unweighted average of the three percentages, rounded to one decimal.
The image does not print how many items each column has, which split was scored, or how the question was asked. The post does not link a repository. Open-Jev’s public slice is a different comparison: Jev 200/231, Luna 206/231, on JevBench tasks. This table is the three Japanese sets. We read the image. We did not rerun the three columns.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Primary post is @Trtd6Trtd, display name t.toda, September 22, 2026, 23:45 UTC. The post says a Japanese benchmark was tried because Jev had been described as not very smart, that general knowledge looks ordinarily strong, that JamC-QA was weak, that academic questions may be fine while fine-grained knowledge specific to Japan is weaker, and that Bonsai 2's one-token logit does not look like a Jev substitute.
The image is a table. Jev: JCommonsenseQA 96.0%, JMMLU 90.2%, JamC-QA 56.0%, mean of the three 80.8%. Bonsai 2 (1-token logit): 93.8%, 66.8%, 41.2%, mean 67.2%. GPT-5.6 Luna: 97.2%, 82.5%, 76.5%, mean 85.4%.
The fourth column is labelled as the average of the three. The image does not print item counts, a split, a prompt, or a repository. We read the image. We did not rerun the sets.
Compare
Open-Jev's public JevBench slice puts Jev at 200/231 and Luna at 206/231, with Jev ahead of Luna on neither the full public slice nor the hard tier. This table has Jev ahead of Luna on JMMLU and behind on JamC-QA and on the three-set mean.
The denominators are not printed, so the percentages cannot be lined up with the 231-task counts.
Terms
- JamC-QA
- The third column on t.toda's chart. Jev is printed at 56.0% and GPT-5.6 Luna at 76.5%. The post suggests this column is where Japan-specific detail shows up. Item count is not on the image.
- three-set mean
- The fourth column. It is the average of the three printed percentages. Jev 80.8%, Bonsai 2 67.2%, Luna 85.4%.