TypeLLM scores Qwen3.8-27B at 228 of 231 public JevBench tasks when thinking is on

TypeLLM is a Python library, and a hosted API, that keeps an autoregressive model and constrains each field to a JSON Schema. The September 17 introduction says the project was prompted by TypeSafe’s Jev. Jev is a purpose-built decision model. TypeLLM’s pitch is the open models a caller already serves. The library is on GitHub. PyPI typellm 0.6.8 was uploaded at 08:41 UTC on October 9, 2026. The license expression is Apache-2.0. It requires Python 3.10 or newer. The project owner field reads rejojer. The pages we opened do not give that account a personal name.

The homepage table is the September 23 self-hosted run. The file scores RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead on 231 public JevBench tasks. No thinking: 195 correct. Thinking: 228 correct. The homepage prints 84.42% and 98.70%. Those are 195/231 and 228/231 at two decimal places. The method note says this is one run per setting, on the public subset, and not a full 534-task board score. Schema validity is 231 of 231 in both files. We did not run verify.py.

The 33-task gap is the hard tier. Hard goes from 76 of 111 to 109 of 111. Easy goes from 48 of 48 to 47 of 48. Original goes from 71 of 72 to 72 of 72. Those three movements net to 33, which is 228 minus 195. The method splits the 231 tasks into 72 Original, 48 Easy, and 111 Hard, and into 139 Choice, 74 Noul, and 18 Score questions. The upstream pin is f8ce71361165846101d02ebc83ad44e47ae44fc3.

Four families in the same files supply 25 of the 33. Temporal numeric goes from 8 of 15 to 15 of 15. Probability goes from 4 of 10 to 10 of 10. Multi-hop goes from 12 of 18 to 18 of 18. Long policy goes from 12 of 19 to 18 of 19. Extraction goes the other way, from 24 of 24 to 23 of 24. The file does not say whether that lost extraction item is the lost easy item.

The three thinking misses sit in the high bin. The thinking summary’s bin from 0.9 to 1.0 holds 225 tasks, and 222 of them are correct. The printed accuracy on that bin is 0.9866666666666667. The other bins in that run, two tasks from 0.5 to 0.6 and four from 0.8 to 0.9, are all correct. The three errors are among the answers whose top probability was at least 0.9. Mean confidence in that bin is 0.9976.

Brier falls from 0.240853 to 0.029395. Top-label ECE falls from 0.052494 to 0.016979. Score ordinal MAE falls from 0.272135 to 0.000002. Paraphrase agreement is 35 of 36 pairs without thinking and 36 of 36 with it. The method says accuracy uses the highest-probability label, and that the typed score is the probability-weighted level. The 18 score questions are mapped to an integer enum. There is no permutation averaging in this run.

Latency moves the other way. Median request time goes from 1.296 seconds to 6.677 seconds, about 5.2 times. The 95th percentile goes from 2.150 seconds to 61.083 seconds, about 28 times. The method says those clocks include the client and an SSH tunnel, and that the thinking run reused a warm cache. Thinking tokens in the scored responses total 212,255. Divided by 231, that is 918.9 tokens a task, the average the eval README prints. The run set thinking_budget to null. All 231 thinking answers closed on their own. The local seed of 42 does not seed the server-side reasoning. Temperature on that reasoning is 0.6, with top-p 0.95 and top-k 20. Classification itself is argmax.

The homepage also prints Jev 1.13.0 at 86.58%, Open-Jev 27B v1.1 at 85.28%, GPT-5.6 Luna at 89.18%, and GPT-6 Astra at 100.00%. At two decimal places those four percents are 200, 197, 206, and 231 of 231. The eval README says the external rows come from Open-Jev and that the settings differ. Open-Jev’s row in that file is the same Qwen3.8-27B base with a rank-8 LoRA and a decision head, in BF16. TypeLLM’s row is NVFP4 weights, a BF16 language-model head, and no extra training. We did not open the Open-Jev page. This site’s JevBench composite, and Drex’s 199 of 231, are other tables.

The hosted API names a different model. Docs say it serves typellm-latest, which reads text and images and can think. The caller can leave model out. GET /v1/models is documented as returning that one name. The September 23 checkpoint name does not appear on that page. We did not call https://api.typellm.ai/v1/generate.

The homepage prices input at $0.05 per million tokens and thinking at $0.50 per million. Typed answers are free. New users get $5. At those rates, and with no thinking tokens, $5 pays for 100 million input tokens. With no input tokens, $5 pays for 10 million thinking tokens. That is rate-card arithmetic, not an invoice. TypeSafe’s models page, in the October 7 reading on this desk, lists $0.042 per million input tokens. $0.05 is 1.19 times that figure. The two cards are separate. The 212,255 thinking tokens in the September 23 file, priced at $0.50 per million, come to $0.106. The summary marks the run self-hosted and unmetered, so $0.106 was not charged there.

Hosted limits on the API page, opened October 10: 64 questions a call, 128K tokens of input, a 128 MB body, eight images, and string answers capped at 128 tokens. A thinking budget defaults to 1,024 tokens and may not exceed 4,096. The September 23 run used no budget, so its 228 of 231 is not a measurement of that cap. Timeout defaults to 60 seconds and may not exceed 90. The account rate is 200 requests a minute, shared by its keys. The playground line says 20 a minute before an access code. The library README says a self-hosted caller can pass text_max_tokens for a longer string. The output-types page calls 128 tokens a fixed service limit. Those two sentences describe different places to run.

@TypeLLM posted on September 29 that the playground was open with $5 of credit, and that the API was rolling out to early-access users. On October 7 the same account posted that OpenAI Decisions code runs on TypeLLM after the caller changes the API key and the base URL. The API reference says POST /v1/decisions uses base https://api.typellm.ai/v1, and POST /v1/systemone uses base https://api.typellm.ai. Both are answered and billed as /v1/generate. We did not send either call.

The docs models page lists five self-host rows: Qwen3.8-27B, Qwen3.5 at 0.8B, 4B, and 9B, MiniCPM5-1B, Ling-mini-2.0 with thinking off only, and Ring-mini-2.0 with thinking always on. Image input is tested with Qwen3.8-27B. The GitHub README’s supported-models table, read the same day, names the Qwen rows and does not name MiniCPM, Ling, or Ring. The README’s comparison table says rubric scoring is a numeric enum and that TypeLLM has no dedicated Score API. Further up, the same README documents a levels field that returns the probability-weighted index of the levels. The output-types page has a Score row with the same shape. Both sentences were on the pages we opened.

A field can name depends_on. The child then sees the parent’s typed answer, and a cycle is rejected before inference. The value is still only guaranteed to land inside the child’s own domain. A wrong parent is an input to the child, and the schema does not correct it. when tests those typed answers in code and adds no model request. A failed test skips the field. The README’s confidence formula is (p_max - 1/n) / (1 - 1/n). On the printed three-way example, probabilities 0.04, 0.93, and 0.03, that formula returns 0.895. The README prints confidence 0.9.

The September 22 die post is a different measurement from the 231 tasks. One Jev call, options written one through six, puts 83% on one. Across all 720 orderings, Jev’s highest probability stays on the label one every time, and lands on the first option 120 times. 120 of 720 is 1/6, which is how often one sits in front if the winner is always that label. TypeLLM plus Qwen, thinking off, puts the highest probability on the first option in 718 of 720 orderings, and on the label one in 122 of 720. The two counts are each two away from the “always the label” pair of 720 and 120. The post does not print those two orderings, so the overlap is a count check. After averaging all 720 orders, Qwen’s one is 25.41% and Jev’s one is 86.01%, against a fair 16.67%. Eight sampled orders, the post says, cut Qwen’s KL error by 79% relative to its average single-order error. All 720 cut it by 97%. Jev moved only slightly. The script named in that post expects the same NVFP4 checkpoint. We did not roll the die.

Two letter-count traces are also different prints. The thinking docs show 68 thinking tokens for strawberry. The example page shows 157 input tokens, 123 thinking tokens, 0.84 seconds, and a cost of $0.000069. At $0.05 and $0.50 per million, 157 input tokens plus 123 thinking tokens come to $0.00006935, which rounds to the printed cost. The 68-token trace is a different line.

Other hosted calls of this shape are on decision models like Jev.