Published
Updated
jev_local's Sarashina row beats LFM on two JGLUE tests, and the README says that is not a ranking
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
あるごす added Sarashina2.2 Vision 3B to a local /v1/systemone server. On JGLUE test the Q4 checkpoint is 34% on JNLI and 86% on JCommonsenseQA, against LFM at 17% and 69%. ModernBERT-Ja, trained on the JGLUE train split, is 93% and 92%. The post times the 3B model at about 100 ms when it is fast and about 500 ms as the usual case, on a Radeon AI Pro R9700.
あるごす posted on September 23 that Sarashina2.2 Vision 3B is now a backend in jev_local. The post says the mradermacher GGUF on Hugging Face had a broken image path, so that piece was rebuilt and published, and that llama.cpp lacked Sarashina’s FlashAttention, so a patched build is on GitHub. At 3B, a fast answer is still 100 ms, and about 500 ms is the usual case, on a Radeon AI Pro R9700.
The README serves the same POST /v1/systemone from three backends. LFM2.5 is the default, a 1.2B instruct model or a 1.6B vision model at Q8_0, reading the first-token logprob. Sarashina2.2 Vision 3B uses the same read, at Q4_K_M (about 2.07 GB) or Q8_0 (about 3.57 GB). ModernBERT-Ja 310M is a fine-tuned cross-encoder over the pair (question plus state, candidate). The JGLUE test row on that page: LFM is 17% on JNLI and 69% on JCommonsenseQA. Sarashina Q4 is 34% and 86%. ModernBERT is 93% and 92%, and the README says that row was trained on the JGLUE train split. The same paragraph says the training conditions and the quantization differ, so the table is a practical comparison on one API, not a ranking of the models.
docs/SARASHINA.md adds the limits. JNLI, an entailment set, falls below a baseline that always picks the majority class. Choice stops at 26 candidates. Twenty-seven or more returns 422. LFM and ModernBERT on this server still accept 255. Q8 is slightly higher on commonsense QA and knowledge questions, and slightly faster on short decisions. JNLI is higher at Q4. Reversing the candidate order can change the decision. Probabilities are not calibrated. Response time is about 2.5 to 4 times LFM. The 100 ms and 500 ms figures are in the post, for a Radeon AI Pro R9700. The README does not print that pair.
Image input needs a separate mmproj and a patched runtime. The CUDA build of that runtime is not verified on NVIDIA hardware. Solid-color images and questions about the order of two or more images still fail. Text-only run.sh --model sarashina returns 502 if an image is sent. The repository has no LICENSE file. The README says the server is an unofficial adapter to TypeSafe’s HTTP shape, not a copy of Jev’s weights or training. We did not run it.
A Japanese table put hosted Jev at 96.0% on JCommonsenseQA and printed no item count. This README’s JCommonsenseQA cells are local, and there is no Jev column on them. ModernBERT’s 92% was fit on the train split of the same suite.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Primary post is @argos_M1111, September 23, 2026, 03:05 UTC. Display name あるごす. Text: a domestic multimodal local Jev was added, Sarashina2.2 Vision 3B. The mradermacher GGUF on Hugging Face had a broken image-input path, so that file was rebuilt and published.
llama.cpp did not support Sarashina's FlashAttention, so a patched build is on GitHub. At 3B, even a fast response is 100 ms, and about 500 ms is typical, on a Radeon AI Pro R9700.
README comparison, JGLUE test: LFM JNLI 17% / JCommonsenseQA 69%; Sarashina Q4 34% / 86%; ModernBERT JNLI 93% / JCommonsenseQA 92%, and that row was trained on the train split. The README says training conditions and quantization differ, so the table is a practical comparison on one API and not a fair ranking.
SARASHINA.md: JNLI falls below a majority-class baseline. Choice is capped at 26. 27 or more returns 422. Q4_K_M is about 2.07 GB, Q8_0 about 3.57 GB. Q8 is slightly higher on commonsense QA and knowledge and slightly faster on short decisions; JNLI is higher at Q4.
Response time is about 2.5 to 4 times LFM. The 100 ms and 500 ms pair is the post, not a cell in the README. Option order can flip an answer. Probabilities are not calibrated. The CUDA Sarashina build is not verified on NVIDIA hardware.
The repository has no LICENSE file. The README says this is not a reproduction of Jev's weights or training. We did not run the server.
Compare
The Japanese table posted on September 22 scores hosted Jev at 96.0% on JCommonsenseQA, with no item count. This README scores local backends on JGLUE test and does not include a Jev column. ModernBERT's 93% and 92% were fit on the train split.
Sarashina's 34% on JNLI is above LFM's 17% and, on the author's note, still under a majority baseline.
Terms
- JGLUE test row
- The README's shared-API table. LFM: JNLI 17%, JCommonsenseQA 69%. Sarashina Q4: 34% and 86%. ModernBERT-Ja 310M, trained on the JGLUE train split: 93% and 92%. The page says this is not a fair ranking of model ability.
- 26-candidate cap
- Sarashina Choice rejects 27 or more options with HTTP 422. The same README says LFM and ModernBERT still accept 255. Jev's documented cap is 255.