Published
SemIf, a 4B open rebuild that reads option logits, sits 0.7 behind Jev on JevBench
Theodore Lee's SemIf (formerly OpenJev) scores typed options from a frozen Qwen3.5-4B by reading logits, not by sampling JSON. On JevBench v1.2.2 it is 74.6 against Jev 1.13.0 at 75.3. The author's own 3090 timings put direct readout at 1.023 s for 21 questions against 5.332 s for a compact JSON array.
Theodore Lee posted an open typed-decision stack in mid-September under the name OpenJev. The repo is now TheoLeeCJ/SemIf. The README says it is not affiliated with TypeSafe. Florian S’s JevBench table is what put a number next to hosted Jev: 74.6 against 75.3 for jev-1.13.0.
SemIf does not sample an answer. Criteria and option strings arrive with the request. A frozen Qwen3.5-4B (revision pinned in the README) scores those options from native logits. Shared state can be prefilled once and branched. A WebGPU demo runs quantized GGUF in the browser.
The author’s 3090 note, same weights, same 21 binary criteria, median of three: direct readout 1.023 s and zero output tokens; a compact yes/no JSON array 5.332 s and 111 tokens. First token on the generative path was 0.489 s; finishing the array took 5.21 times as long as the logit read. The three arrays were valid and identical. They matched direct argmax on 18 of 21 criteria, so the README treats this as a systems comparison.
On an owned 37-state by 21-criterion fixture, fresh scoring ran at 2.33 decisions per second, serial prefix reuse at 10.75, parallel suffixes at 20.03. The fast paths changed 5 to 6 of 777 argmaxes relative to fresh BF16 scoring.
Quality, from the repo: authored balanced accuracy 0.813 on 144 rows. On 102 TypeSafe public rows the authors could align, modal agreement 0.845 against a published Jev figure of 0.883. They did not call a live Jev endpoint.
JevBench is the independent row. Hard-tier accuracy there is 59.5% for SemIf and 74.1% for Jev. Cost is an estimate, about $0.023 per 1,000 decisions. Latency is from a rented GPU with the bench’s ×2 plus 0.15 s adjustment. We did not run the scorer.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
JevBench v1.2.2 lists "SemIf, formerly OpenJev (Qwen3.5-4B, TheoLeeCJ)" at 74.6, Intelligence 85.9, Calibration 72.6, Speed 83.7, Cost 59.2 (~$0.023 estimated per 1,000 decisions), hard-tier accuracy 59.5% against Jev 74.1%. The GitHub repo is TheoLeeCJ/SemIf; older links to TheoLeeCJ/openjev redirect there. The README states it is independent and not affiliated with TypeSafe. Direct readout on a frozen Qwen3.5-4B, same 21 binary criteria, RTX 3090, median of 3: 1.023 s and 0 output tokens versus 5.332 s and 111 tokens for a compact JSON array (5.21×). Choices agreed with direct argmax on 18/21. On an owned 37×21 fixture, parallel suffixes ran at 20.03 decisions/s. Authored balanced accuracy 0.813 (144 rows). TypeSafe selected-subset modal agreement 0.845 on 102 aligned rows against a published Jev figure of 0.883; the authors did not call a live Jev endpoint. Browser WebGPU demo ships quantized GGUF. We did not run the scorer. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; the JevBench row is reported as published.
Compare
jeff copies POST /v1/systemone on GLiFormer and trails Jev on public labels (JevBench 66.9). LocalJev prompts a local model for JSON probabilities and says that is not a logit read; its README's "OpenJev" is the DiffusionGemma structured-read path (razorback16 on JevBench at 67.6), a different repo. Nimble is a 9B LoRA scored on its own 324 labels. SemIf is the closest open row on JevBench, 0.7 behind Jev on the composite and 14.6 points behind on the hard tier. The 5.21× JSON timing is the author's 21-question fixture on one 3090, not JevBench's latency column.
Terms
- SemIf
- Theodore Lee's open project, formerly OpenJev, that reads typed option probabilities from a local model. JevBench v1.2.2 reports 74.6 for the Qwen3.5-4B row.
- Direct logit readout
- One forward pass over declared option tokens, with no sampled answer. SemIf's 3090 note: 1.023 s for 21 questions versus 5.332 s for a compact JSON array from the same weights.