llamacpp-jev answers /v1/systemone from an unmodified llama-server, including images

Chirag posted llamacpp-jev on September 22. The note says a comment claimed any local model could speak TypeSafe’s /v1/systemone, so he built a server that does that in front of an unmodified llama-server. The GitHub account on the link is NakliTechie. Package metadata is llamajev 0.1.0, MIT, Python 3.12 or newer, and the pyproject.toml does not list an author name.

The wrapper renders the model’s own chat template, warms one prefix, then asks one token per question and reads logprobs for labels A, B, and then two-letter labels once the alphabet runs out. Choice and score take 2 to 64 options. A request holds 1 to 64 questions. The grammar forces the sampled token to be one of those labels. The probability vector comes from the pre-sampling logprobs, then a softmax over just the labels. docs/DESIGN.md says those probabilities are not calibrated, and that confidence is 1 - H / log(n), the formula used by openjev-sglang. The same page says TypeSafe has not published its confidence formula. usage.input_tokens adds up llama-server’s prompt counts, including tokens served from cache.

Images go in as image_url data URIs. That path needs --mmproj. The design table lists images as absent from TypeSafe’s API and from openjev-sglang. llama.cpp was checked at master 3d82ef62, build b11063, dated 2026-09-20. The default server line turns the RAM prompt cache off (--cache-ram 0), keeps context checkpoints, and pins each request to one slot so the image is encoded once.

The README’s September 22 timing, Qwen3.5-2B-Q8_0 on an M4 Pro with 24 GB, Metal, GPU otherwise idle, four slots:

• Four typed questions on a never-seen 448×448 image, eight images: median 526 ms, range 525 to 546, 32 of 32 correct
• The same image again: 254 ms
• Four text questions, repeated: 238 ms

A September 21 pass on that Mac, while another job used the GPU, recorded 946 ms (826 to 1048) for the fresh image and the same 32 of 32. The design notes say the later 526 ms is partly a quieter GPU, not a proven speedup in the code. An idle NVIDIA L4, same weights and the same llama.cpp commit, recorded 236 ms (174 to 237) and 32 of 32 on the fresh image, 180 ms on repeated text. Prefill on the L4 was 83 ms against 292 ms on the M4 Pro.

Correctness beyond the shapes: six Wikimedia photos, each scored by hand, 22 of 22 answers the author was willing to mark, with two left out as ambiguous. Those photo runs took 1 to 5 seconds because a second server was on the GPU, so the notes leave them out of the speed table. The README calls the accuracy check synthetic shapes plus that spot check, not a benchmark.

Both Qwen3.5-0.8B and 2B got 4-way, 10-way, and 26-way questions right at the positions tried, including option 20. On 64-way questions both answered Q, which is option 16, wherever the correct option was. The notes leave the cause open. The HTTP contract still accepts 64 options.

--cache-ram 4096 stalled inside llama-server on Metal while copying checkpoint state. The default is --cache-ram 0 until that is understood. The same repro on an idle L4 did not stall. The slowest call there was 299 ms. Concurrent load on CUDA was not part of that run.

The README points at a September 20 post from @kis that described about 800 ms for four questions on a 448 image and did not link a repository. This repo is a separate build of that shape.

That post is 21:43 UTC on September 19. In Japan that clock is the morning of September 20, the date the README prints. The text has no URL. The attachment is a video.

At 22:37 UTC the same account replied with the jev branch of kishida/llama.cpp. The card on the reply is docs/jev.md. The next reply says the web UI opens at localhost:8080/jev.

That page says you build the jev branch because the endpoint is not in upstream llama.cpp yet. The binary is llama-server. It serves POST /v1/systemone and GET /jev. Each question is one prompt. The probabilities are the next-token logits of the labels, and output_tokens is 0. confidence is 1 minus normalized entropy. Labels are single tokens from A–Z, a–z, and 0–9, usually at most 62. Images need --mmproj. Eight questions about one photo, on Qwen3.5 2B, are 1.6 s in one request and 4.6 s when the image is encoded again for each question.

llamajev still starts an unmodified llama-server and reads logprobs beside it. The fork compiles the HTTP path into llama-server.

The reference table is 1,191 multiple-choice questions, 2 to 8 options. Guessing scores 0.283. Temperature is fit on a separate 358-question split. Most rows are an RTX 4060 Ti, -ngl 99 -c 16384 -np 8, from scripts/bench_models.py of the Jev PoC. Rows marked Mac are another machine. The page says to compare their accuracy and not their time. Time is the mean of 50 sequential requests with one question each.

Opened October 3, Qwen3.8 Flash-Next, UD-Q4_K_XL, is 0.944, ECE 0.023 to 0.015, temperature 1.26, 292 ms, marked Mac. Qwen3.6 35B A3B is 0.915 at 101 ms. Gemma 4 26B A4B is 0.913 at 92 ms. Both of those are marked Mac, and both are mixture-of-experts. GLM-5.3-Flash is 0.917 at 692 ms with --jev-assistant-prefix set to a newline, </think>, and a newline, and 0.541 without that prefix. The latest commit message on docs/jev.md in that history is October 1, “docs: jev, GLM-5.3-Flash is upstream now.”

On October 2 @kis posted a screenshot of a shorter table and wrote that Qwen3.8-Flash-Next has the best result and that the table does not include it. The screenshot’s remote rows match the doc: Gemma 4 31B it qat 0.935 at 418 ms, Qwen3.6 27B 0.923 at 403 ms, Qwen3.6 35B A3B 0.915 at 101 ms, Gemma 4 26B A4B 0.913 at 92 ms.

Qwen3.5 on the doc, four sizes: 2B Q8_0 is 0.709 at 52 ms, 4B Q4_K_M is 0.852 at 101 ms, 9B is 0.885 at 116 ms, 27B UD-IQ3_XXS is 0.914 at 362 ms. Three 27B generations at that quant, Qwen3.5 0.914, Qwen3.6 0.919, and Qwen3.8 0.914, sit within 0.005. The page calls that noise. The 52 ms row is one text question. It is not the four-question image timing.

Pull 29818 on ggml-org/llama.cpp merged October 2. The title adds /v1/systemone for Laya, Julia-1, lev, OpenJev, and Kev. The Hugging Face note the same day, by Xuan-Son Nguyen and Victor Mustar, says llama-server now answers that path for those decision-model GGUFs. A client keeps its request and changes the base URL. The median time for one question on one NVIDIA RTX PRO 6000 is 3 ms for Julia-1 (144M, mmBERT-small, Apache 2.0), 5 ms for Laya (421M, ModernBERT-large, Apache 2.0), 12 ms for Kev-4B (Qwen3.5-4B-Base, Apache 2.0), 36 ms for lev (Qwen3.5-4B, Apache 2.0), and 43 ms for OpenJev (27B, Qwen3.8-27B, images, CC BY-NC 4.0).

Score on that page takes 2 to 10 levels. The sample response rounds a billing choice to 0.9049 with confidence 0.8574, a noul to 0.8208, and a score to 2.2821, with output_tokens at 0. Julia-1 sent “I was charged twice” to shipping on bare labels and to billing at 0.99 once the options had descriptions. A vague ticket scored 0.25 on Julia-1 and 0.80 on Kev-4B. The note says Cloudflare’s Clef is next. The branch doc still said, on October 3, that the endpoint is not in upstream llama.cpp yet. David Hendrickson’s October 2 post states the same five GGUFs and puts the blog URL in the image alt text.

Release v0.6.0 was published October 5, 2026, at 16:56 UTC. The nightly build named on that page is b11429. The overview lists /v1/systemone for laya, julia-1, lev, openjev, kev, and nimble, and lists Clef for text and for images. The highlights still describe pull 29818 as five models.

Pull 29844 merged October 2 at 12:56 UTC. It adds the nimble decision type for Bespoke-Nimble-9B-v3. That card says the checkpoint is a LoRA on Qwen3.5-9B, prints Decision Index 0.2.1 at 56.88, and says the license is CC BY-NC 4.0. The base model on the card is Apache 2.0. The card describes a choice among options, or a yes or no. 56.88 is that card. It is not the 324-label holdout on Bespoke Nimble.

Pull 29831 merged October 3 at 00:50 UTC. The commands in that pull are llama serve -hf ggml-org/Clef-GGUF and llama serve -hf ggml-org/Clef-Flash-GGUF. The pull said images were still missing. Pull 29969 merged October 5 at 12:02 UTC and adds image input for Clef.

The server README at tag v0.6.0 says OpenJev and Clef are the image examples. A request with an image returns 501 when the model cannot read images, or when no multimodal projector is loaded. A model that is not a decision model also returns 501. A choice is limited to 52 options on OpenJev, and to 255 on Laya and Clef. Questions in one request are answered separately, except Clef, which reads all of them in one prompt. A server that loaded Clef serves only /v1/systemone. In server-decision.h at that tag, image input is OpenJev and Clef. OpenJev, lev, Kev, and Nimble can reuse a shared prompt prefix. Laya does not.

Pull 29903, merged October 3, sets n_batch equal to n_ubatch for these models. The pull says Laya turns embedding mode on after the model loads, and a larger logical batch then aborts.

Opened October 6, the Hugging Face note has six rows. Clef is 27B, based on Qwen3.8-27B, English, images, Apache 2.0, and the speed cell is blank. The images section on that page still names OpenJev as the image model at the time of writing. The closing section no longer says Clef is next. Nimble is not in the table. The same day, docs/jev.md on the kishida branch still says the endpoint is not in upstream llama.cpp. The newest commit on that file is still October 1, “docs: jev, GLM-5.3-Flash is upstream now.”

We did not install llamajev or repeat the timings. We did not build the jev branch or run the upstream server.