Published
llamacpp-jev answers /v1/systemone from an unmodified llama-server, including images
Chirag posted llamacpp-jev, an MIT wrapper that puts TypeSafe's decision API in front of stock llama-server. On an M4 Pro, four questions about a fresh 448 by 448 image took 526 ms median, 32 of 32 on synthetic shapes. The probabilities are a raw label softmax. Qwen3.5-2B picks option 16 on every 64-way question the notes tried.
Chirag posted llamacpp-jev on September 22. The note says a comment claimed any local model could speak TypeSafe’s /v1/systemone, so he built a server that does that in front of an unmodified llama-server. The GitHub account on the link is NakliTechie. Package metadata is llamajev 0.1.0, MIT, Python 3.12 or newer, and the pyproject.toml does not list an author name.
The wrapper renders the model’s own chat template, warms one prefix, then asks one token per question and reads logprobs for labels A, B, and then two-letter labels once the alphabet runs out. Choice and score take 2 to 64 options. A request holds 1 to 64 questions. The grammar forces the sampled token to be one of those labels. The probability vector comes from the pre-sampling logprobs, then a softmax over just the labels. docs/DESIGN.md says those probabilities are not calibrated, and that confidence is 1 - H / log(n), the formula used by openjev-sglang. The same page says TypeSafe has not published its confidence formula. usage.input_tokens adds up llama-server’s prompt counts, including tokens served from cache.
Images go in as image_url data URIs. That path needs --mmproj. The design table lists images as absent from TypeSafe’s API and from openjev-sglang. llama.cpp was checked at master 3d82ef62, build b11063, dated 2026-09-20. The default server line turns the RAM prompt cache off (--cache-ram 0), keeps context checkpoints, and pins each request to one slot so the image is encoded once.
The README’s September 22 timing, Qwen3.5-2B-Q8_0 on an M4 Pro with 24 GB, Metal, GPU otherwise idle, four slots:
- Four typed questions on a never-seen 448×448 image, eight images: median 526 ms, range 525 to 546, 32 of 32 correct
- The same image again: 254 ms
- Four text questions, repeated: 238 ms
A September 21 pass on that Mac, while another job used the GPU, recorded 946 ms (826 to 1048) for the fresh image and the same 32 of 32. The design notes say the later 526 ms is partly a quieter GPU, not a proven speedup in the code. An idle NVIDIA L4, same weights and the same llama.cpp commit, recorded 236 ms (174 to 237) and 32 of 32 on the fresh image, 180 ms on repeated text. Prefill on the L4 was 83 ms against 292 ms on the M4 Pro.
Correctness beyond the shapes: six Wikimedia photos, each scored by hand, 22 of 22 answers the author was willing to mark, with two left out as ambiguous. Those photo runs took 1 to 5 seconds because a second server was on the GPU, so the notes leave them out of the speed table. The README calls the accuracy check synthetic shapes plus that spot check, not a benchmark.
Both Qwen3.5-0.8B and 2B got 4-way, 10-way, and 26-way questions right at the positions tried, including option 20. On 64-way questions both answered Q, which is option 16, wherever the correct option was. The notes leave the cause open. The HTTP contract still accepts 64 options.
--cache-ram 4096 stalled inside llama-server on Metal while copying checkpoint state. The default is --cache-ram 0 until that is understood. The same repro on an idle L4 did not stall. The slowest call there was 299 ms. Concurrent load on CUDA was not part of that run.
The README points at a September 20 post from @kis that described about 800 ms for four questions on a 448 image and did not link a repository. This repo is a separate build of that shape.
We did not install llamajev or repeat the timings.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Primary post is @chirag on September 22, linking github.com/NakliTechie/llamacpp-jev. Package llamajev 0.1.0, MIT, Python >=3.12. pyproject.toml has no authors field. The server is POST /v1/systemone in front of an unmodified llama-server, verified at llama.cpp master 3d82ef62 (build b11063, 2026-09-20). Questions are noul, choice, and score, 1 to 64 questions, 2 to 64 options. Labels are single tokens (A, B, then AA). Probabilities are a softmax over those label logprobs, read before grammar sampling. The README says they are not calibrated. confidence is 1 minus normalized entropy, the openjev-sglang definition. DESIGN.md says TypeSafe does not publish its own confidence formula. usage.input_tokens sums backend prompt counts and includes cached tokens. Images are image_url data URIs and require --mmproj. Default llama-server flags include --cache-ram 0, --ctx-checkpoints 32, and slot pinning. Measured 2026-09-22 on Qwen3.5-2B-Q8_0, M4 Pro 24 GB, Metal, GPU uncontended, 4 slots, pinning, cache-ram 0: four typed questions on a never-seen 448 by 448 image, 8 images, median 526 ms (525 to 546), 32/32 correct. Same image repeated, 254 ms. Four text questions repeated, 238 ms. The September 21 row on the same host, with another job on the GPU, was 946 ms (826 to 1048) for the fresh image, still 32/32. DESIGN.md says that drop is partly load. An idle NVIDIA L4, same model and llama.cpp commit, fresh image 236 ms (174 to 237), 32/32; text repeated 180 ms. Six Wikimedia photos, hand-labelled, 22/22 scored answers, two left unscored as ambiguous. Photo latencies were 1 to 5 seconds because a second server shared the GPU, so they are not in the speed table. Both Qwen3.5-0.8B and 2B answer 4, 10, and 26-way questions at the positions tested, and on 64-way questions both choose Q, option 16, no matter where the right option sits. Cause not established. With --cache-ram 4096 the Metal notes record a checkpoint stall inside llama-server. An idle CUDA run of the same repro did not stall (slowest call 299 ms). Concurrent CUDA load was not tested. The README traces the idea to @kis on September 20, a claim of about 800 ms for four questions on a 448 image, with no repository linked. We did not run llamajev.
Compare
SimpleJev and ekzhang/openjev-sglang also turn label logprobs into the decision JSON. LocalJev prompts a local model for JSON probabilities and says that is not a logit read. jeff is a separate encoder. Hosted Jev, on TypeSafe's models page, takes text. This wrapper accepts an image when llama-server was started with a multimodal projector. The 526 ms figure is four questions on one laptop image, not a JevBench row. The 32/32 and 22/22 counts are synthetic shapes and six hand-labelled photos.
Terms
- llamajev
- The MIT command in NakliTechie/llamacpp-jev. It launches or attaches to an unmodified llama-server and serves POST /v1/systemone by reading single-token label logprobs.
- raw label softmax
- llamajev's probabilities are a softmax over the logprobs of the option labels. The README says they are not TypeSafe's calibrated RLCD probabilities.