rev reads a frozen Qwen and posts 0.784 on public hard items

Steven Yu posted 54yyyu/rev on September 22. The repository is MIT. It takes a state, a question, and a closed list of options, and it returns a probability for each option from one forward pass. rev serve answers POST /v1/systemone on 127.0.0.1:8421. The default local checkpoint is Qwen/Qwen3.5-2B at 8-bit, about 1.86 GB of weights, on Apple Silicon through MLX. A second command, rev serve --upstream, leaves the weights on an sglang server and reads the log-probability of each answer letter.

The public-tier table is dated 2026-09-22. Qwen3.8-27B, INT4 AWQ, two L40S cards, exact logprobs: easy 1.000 (48 items), standard 0.986 (72), hard 0.784 (111). Thirty-two samples land on the same three cells. Qwen3.5-2B in-process on an M2 Pro with 16 GB is 1.000 / 0.764 / 0.550. Jev 1.13.0 on that page is 1.000 / 0.986 / 0.730. The hard cell also prints 0.721, measured the same day. The README says the 27B row is not on the public leaderboard, and it points at docs/JEVBENCH.md for why. On the family it calls temporal_numeric, the 27B note is 0.400 and Jev is 0.267. At a 0.9 confidence gate the 27B note answers 78% of the standard items, all of those correct, and 35% of the hard items, also all correct.

A second 2B table, same laptop, puts frozen rev at hard 0.550, a single option order at 0.523, a fine-tune called decider-2b at 0.477, and a published reflex 2B figure at 0.468, with Jev still at 0.730. The README says the harness started as a way to compare SemIf, reflex, and decider, and that a frozen Qwen3.5-2B moved from 0.428 to 0.550 on the hard public items without a weight change. It also says 8-bit instead of 4-bit is about nine points on hard, at no latency cost, and that 8-bit and bf16 gave the same answers. A fitted per-option offset is listed at +7.9 points on binary questions.

Speed on the 27B path includes two network hops: the laptop talks to a host, and the host talks to the cluster over an ssh relay. Median, exact logprobs: a short clipboard paste 145 ms, a standard item 138 ms, a hard item 317 ms, a hard item with three questions 725 ms. Jev on those same rows: 354, 356, 326, and 328 ms. Eight requests in flight, short items: 18.3 per second against Jev at 21.3. The local 2B server, measured 2026-09-21 on an M2 Pro, is faster than Jev on short and medium items (266 ms and 240 ms against 354 ms and 356 ms) and slower on hard items (821 ms against 326 ms). Eight in flight, the laptop does 0.7 to 3.9 per second. Jev does 21 to 24.

Three tasks written as a user would write them, 8-bit, local: clipboard paste 160 items at 1.000, a calendar pick across English and Chinese 216 items at 0.986, a write-action gate 57 items at 0.965. The README says Jev scored 1.000 on the first two, and that every miss on the gate was a safe request sent for confirmation. Mail from one inbox, 146 messages, 19 marked important before the model ran: AUC 0.941, interval 0.89 to 0.98, with 18 of the 19 in the top 18%. A sender regex on the same mail scored 0.375.

The limits section says nine of nine temporal_numeric items were wrong locally, and that one yes/no about whether an email needs an action scored AUC 0.176 on real mail, the reverse of a category question that scored 0.878 on the same messages. An API key placed in the option list came back as an email address, a phone number, and a social profile, each at probability 1.00. “Tidy up my calendar” was classed as a local note edit at 0.99. We did not run the bench.