Published
Bespoke Nimble is an open 9B Jev-style model: 90% on its own 324-example holdout
Bespoke Labs released Nimble, a Qwen3.5-9B LoRA trained on contrastive synthetic pairs, plus the data and recipe. On 324 held-out examples it matched 90.12% of the reference labels. Jev 1.13.0 matched 93.21%. The base 9B model matched 66.36%.
Mahesh Sathiamoorthy, CEO of Bespoke Labs, posted Bespoke Nimble on September 18: an open dataset, a Qwen3.5-9B LoRA adapter, and the recipe. The day before he had asked whether he should build an OpenJev. Weights are on Hugging Face. The code is at bespokelabsai/nimble.
The model takes text and a flat schema (choice or true/false) and returns the picked answer plus a probability for each allowed option. It does not write reasoning. Serving reads the prompt, then scores one answer token per question. The authors say they did not distill from Jev. Jev was used to evaluate. Training is cross-entropy on the allowed logits, LoRA rank 16, one epoch, 2,048-token prompt limit.
The training set is 2,676 synthetic examples across ten subject categories. The method they call contrastive data curation writes a pair that differs in one fact. In the README example, only Mira may authorize refunds; change the signer to Noah and the answer flips from true to false. Labels are assigned by code from checked facts. No person has reviewed them.
On 324 held-out examples (162 pairs, six source families) the README reports:
- Gemma 3 270M IT: 93/324 (28.70%)
- Qwen3.5-9B: 215/324 (66.36%)
- Qwen3.8-27B: 275/324 (84.88%)
- Bespoke-Nimble-9B: 292/324 (90.12%)
- Jev 1.13.0: 302/324 (93.21%)
Median time per example, from saved request timings: 106 ms for Nimble on an H100 (120 examples in that row), 444 ms on an M5 Pro with 64 GB, 246.7 ms for the Jev API on the 324. Sathiamoorthy’s thread rounded the H100 figure to 100 ms and said you can run it on a MacBook.
The holdout is narrow. The labels are synthetic. There is no shared System One benchmark, so Nimble could be much worse than Jev on someone else’s test. Fields cannot depend on each other. Prompts over 2,048 tokens are rejected.
Earlier on September 18, Hemant posted Verdict, a 151M ModernBERT encoder with a GLiClass head and an in-browser WebGPU demo. On 337 cases from TypeSafe’s public evaluation set he reports Verdict at 48.1% against Jev at 90.8%. That is a different architecture and a different test from Nimble’s 324.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The GitHub README and the September 18 thread agree on the recipe: LoRA on Qwen3.5-9B, contrastive synthetic pairs, Jev used as an evaluator rather than a teacher. On 324 held-out examples, Bespoke-Nimble-9B matched 292 labels (90.12%), Jev 1.13.0 matched 302 (93.21%), and the untuned 9B base matched 215 (66.36%). The holdout is 162 closely related pairs from six source families. Labels are synthetic and unreviewed by people. Median latency in the saved timings is 106 ms on an H100 (120 examples) and 444 ms on an M5 Pro with 64 GB (324 examples); Jev's API median on the same 324 is 246.7 ms. We did not train or score the adapter. The authors say there is no standard System One benchmark, and Nimble may do worse than Jev on other tests.
Compare
Hemant's Verdict is a 151M ModernBERT plus GLiClass head that runs in the browser. On 337 cases from TypeSafe's public evaluation set he reports Jev at 90.8%, a 26B diffusion model at 88.4%, and Verdict at 48.1%. Nimble is a 9B generative backbone scored on allowed answer tokens. Verdict is a bidirectional encoder in one forward pass. Both are local recreations with their own evals. Neither is a TypeSafe weight release. Almeida later wrote that constrained decoding that simply masks invalid tokens can make a generative model dumber; Nimble still uses parallel scoring of allowed tokens, which is the serving trick Niels Rogge described.
Terms
- Contrastive data curation
- Write two examples that differ in one fact so the correct answer flips. The rest of the context and the question stay the same.
- Parallel constrained decoding
- Read the prompt once, then score the allowed answer tokens instead of generating a string. Nimble's Python code builds the typed result from those scores.
- Reference-label agreement
- Share of examples where the model's top answer matches the synthetic label. The 324-example table is this metric, not calibration against human judges.