Published
Updated
jevos-1b prints 0.815 on 2,000 yes/no questions, against Jev at 0.927
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Federico Elia released feder-cr/jev, a local server called jevos-1b, built with Loris Salsi. The README's image alt is the printed bench: on 2,000 yes/no questions from unseen policies, accuracy is 0.815 for jevos, 0.927 for Jev, and 0.489 for Laya. Short-prompt latency is 54 ms, 344 ms, and 104 ms. The server accepts Noul and returns 422 for Choice and Score. pyproject says MIT. The root listing has no LICENSE file. We did not rerun it.
feder-cr/jev is a local server Federico Elia posted on September 26, built with Loris Salsi. The product name in the README is jevos-1b. The tree we opened has one commit. pyproject.toml says version 0.0.1 and license MIT. That root listing does not contain a LICENSE file. The weights are a GGUF under the tag jevos-1b, quantization Q4_K_M, downloaded with jev download and run through llama.cpp. The posts attach gifs. This page has no still from the thread. We did not start the server.
The process listens on 127.0.0.1:8017 and accepts POST /v1/systemone. Noul is the question it answers. Choice and Score come back as HTTP 422. A criteria field is accepted and not read. Context is 8,192 tokens. The comparison note says Jev’s context is not stated, and Laya’s is 512. The README does not name the base model under the GGUF.
The printed bench is the alt text on the README image. On short prompts, latency is 54 ms for jevos, 344 ms for Jev, and 104 ms for Laya. On long prompts the three numbers are 220 ms, 345 ms, and 449 ms. Accuracy on 2,000 yes/no questions drawn from policies the README calls unseen is 0.815, 0.927, and 0.489. Asking three questions together takes about 165 ms, against about 103 ms for one question. Sample Noul values in the README, 0.9, 0.78, 0.73, and 0.1, are demonstrations. The list of 2,000 questions was not in the files we opened.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
feder-cr/jev, product name jevos-1b. Federico Elia, with Loris Salsi. Posts on September 26. The repository listing we opened has one commit. pyproject.toml says version 0.0.1 and license MIT.
That listing has no LICENSE file. The GGUF release tag is jevos-1b, quantization Q4_K_M, fetched with jev download and run through llama.cpp. The server listens on 127.0.0.1:8017 and accepts POST /v1/systemone.
Noul is the working question type. Choice and Score return 422. Criteria text is accepted and not read. Context is 8,192 tokens. The README says Jev's context is not stated on the comparison, and Laya's is 512.
The only printed bench is the README image's alt text. Short latency: jevos 54 ms, Jev 344 ms, Laya 104 ms. Long latency: 220 ms, 345 ms, 449 ms. Accuracy on 2,000 yes/no questions from unseen policies: 0.815, 0.927, 0.489.
Three questions together are about 165 ms, against about 103 ms for one. Example Noul outputs 0.9, 0.78, 0.73, and 0.1 are demos. The question list was not in the files we opened. The README does not name the base model.
The posts attach gifs, so this page has no still from the thread. We did not run the server or the 2,000 questions.
Compare
Laya on JevBench is 70.1, a different suite from these 2,000 yes/no questions. llamacpp-jev also speaks POST /v1/systemone from llama.cpp, and its README says the probabilities are not calibrated. jevos returns 422 on Choice and Score, so it does not cover the three question types that server does.
The 0.815 is the alt text, not a JevBench cell.
Terms
- 0.815
- jevos accuracy on 2,000 yes/no questions from unseen policies, as printed in the README image alt. Jev on that line is 0.927. Laya is 0.489.
- 422
- The HTTP status jevos returns for Choice and Score. Noul is the question type the server answers. Criteria text is accepted and not read.
- 8,192
- The context length the README prints for jevos, in tokens. The same note says Laya is 512 and does not state a number for Jev.