Bespoke Nimble 9B matches 90% of its own labels and 74.8% of a public suite

Mahesh Sathiamoorthy, CEO of Bespoke Labs, posted Bespoke Nimble on September 18: an open dataset, a Qwen3.5-9B LoRA adapter, and the recipe. The day before he had asked whether he should build an OpenJev. Weights are on Hugging Face. The code is at bespokelabsai/nimble.

What is Bespoke Nimble 9B?

The Hugging Face weights are Bespoke-Nimble-9B, a LoRA on Qwen3.5-9B. Ollama’s model card writes that release as Bespoke Nimble 9B. The code and the recipe are at bespokelabsai/nimble.

The README says the model takes text and a flat schema. Each field is a choice or a true/false question. It returns the picked answer and a probability for every allowed option, and it does not write reasoning. Serving reads the prompt, then scores one answer token per question.

Training is cross-entropy on those allowed logits. The published recipe is LoRA rank 16, one epoch, and 2,676 synthetic examples across ten subject categories. Sathiamoorthy wrote that Jev was used to evaluate and that this release has no reinforcement learning.

The README says the current objective uses hard reference labels, and that those labels are not derived from Jev. Saved Jev probabilities are available for a later soft-target distillation. This release does not use them as the target.

On the 324 synthetic holdout the same file records 90.12% vs 93.21% for Jev 1.13.0, 292 labels against 302.

On September 30 Ollama posted that version 0.35 serves decision models on a local /v1/systemone endpoint. The command in the post is ollama pull nimble. The same thread lists Tev1 4B and Tev1 0.8B from Together AI beside Nimble.

The model card says one request can ask 1 to 64 questions, and that choice and score fields allow 2 to 26 options. It says the license is Apache 2.0. The tag line prints a 256K context window. The notes on the same page say a prompt has to fit in 8,192 tokens.

The September 30 post shows Nimble playing Ollama racer through that local endpoint. The blog is dated September 29 and says Nimble 9B averaged 91 ms per decision in a Pac-Man example on a MacBook Pro with an M5 Max. The card’s highlight says answers come back in under 100 ms on that machine.

Bespoke Nimble vs Jev

On 324 held-out synthetic labels, the README table is 90.12% vs 93.21%. Bespoke-Nimble-9B matched 292. Jev 1.13.0 matched 302. The untuned Qwen3.5-9B matched 215 (66.36%). The labels were assigned by code, and no person reviewed them. The holdout is 162 closely related pairs from six of the ten training categories.

The September 18 post rounded that comparison to 90% for Nimble and 93% for Jev, and 66% for the base Qwen. Sathiamoorthy wrote that there was no standard System One benchmark, and that Nimble might do much worse than Jev on someone else’s test.

Edgar Dyck’s public suite scores revision 594dfdcf at temperature 1.0 on 3,880 human-labeled records across 13 subsets. None of those tasks is one of the ten training categories. The macro average is 74.8% for Nimble and 76.0% for Jev. Counted by record, the figures are 75.9% and 77.3%.

Choice subsets average 81.6% against 82.9%. True/false subsets average 80.2% against 84.6%. The three score subsets average 54.6% against 50.1%, and Nimble leads that group. Five subsets separate the models on an exact McNemar test at p below 0.05. The other eight do not, on this draw.

Ollama’s model card prints a later mean on the same 13 datasets and 3,880 decisions: 75.7% for Nimble 9B. The Jev cell on that table is still 76.0%. The card says the Jev cell is Bespoke Labs’ published API run, not a new call from Ollama.

Tev1 4B is 73.3% on that mean, and Tev1 0.8B is 63.5%. That 63.5% is Tev1’s accuracy. It is a different figure from the JevBench geometric mean.

Together’s own page for that model, and the chat call that returns one letter, are on Tev1.

The JevBench geometric mean this desk checked is 63.5, with Intelligence 78.6 and Speed 82.5. The write-up is JevBench. Florian’s post the next day said 63.7 and rank 17. On September 25, Benchmark Heaven’s Harold posted a later row for Nimble 9B: hard-tier accuracy 65.5%, JevBench Score 60.5, and about $0.166 per 1,000 decisions.

He wrote that the 90.1% versus 93.2% figures are not from that ledger. We did not reopen the board for that reply.

Saved request timings put Nimble’s median at 106 ms on an H100, on 120 examples in that row, and at 444 ms on an M5 Pro with 64 GB. Jev’s API median on the same 324 examples is 246.7 ms. The announcement rounded the H100 figure to 100 ms. Ollama’s 91 ms is the Pac-Man example on an M5 Max.

Jev’s weights are not in this release. Nimble’s adapter is. On September 19 Sathiamoorthy posted one floor plan and wrote that Nimble counted the bedrooms and bathrooms while the base Qwen struggled. He did not score Jev, because Jev has no image input. The current README says the scorer accepts only text.

The model takes text and a flat schema (choice or true/false) and returns the picked answer plus a probability for each allowed option. It does not write reasoning. Serving reads the prompt, then scores one answer token per question. The authors say they did not distill from Jev. Jev was used to evaluate. Training is cross-entropy on the allowed logits, LoRA rank 16, one epoch, 2,048-token prompt limit.

The training set is 2,676 synthetic examples across ten subject categories. The method they call contrastive data curation writes a pair that differs in one fact. In the README example, only Mira may authorize refunds; change the signer to Noah and the answer flips from true to false. Labels are assigned by code from checked facts. No person has reviewed them.

On 324 held-out examples (162 pairs, six source families) the README reports:

• Gemma 3 270M IT: 93/324 (28.70%)
• Qwen3.5-0.8B: 147/324 (45.37%)
• Qwen3.5-4B: 199/324 (61.42%)
• Qwen3.5-9B: 215/324 (66.36%)
• Qwen3.8-27B: 275/324 (84.88%)
• Bespoke-Nimble-9B: 292/324 (90.12%)
• Jev 1.13.0: 302/324 (93.21%)

Median time per example, from saved request timings: 106 ms for Nimble on an H100 (120 examples in that row), 444 ms on an M5 Pro with 64 GB, 246.7 ms for the Jev API on the 324. Sathiamoorthy’s thread rounded the H100 figure to 100 ms and said you can run it on a MacBook.

The holdout is narrow. The labels are synthetic. The September 18 post said there was no standard System One benchmark, and that Nimble might do much worse than Jev on someone else’s test. Fields cannot depend on each other. The original release rejects prompts over 2,048 tokens.

Earlier on September 18, Hemant posted Verdict, a 151M ModernBERT encoder with a GLiClass head and an in-browser WebGPU demo. On 337 cases from TypeSafe’s public evaluation set he reports Verdict at 48.1% against Jev at 90.8%. That is a different architecture and a different test from Nimble’s 324.

On September 20 the repository published data/train.jsonl and data/eval.jsonl. Both had been left out of the first push. The 324 holdout examples cover six of the ten categories: public services 106, education 70, commerce 58, media 44, supply chain 30, and travel 16. Home, science, software, and workplace appear only in training. By question type the holdout is 146 choice, 114 true/false, and 64 score.

Edgar Dyck’s public suite, added in the repository on September 18, scores Bespoke-Nimble-9B at revision 594dfdcfb6f94e3d0c0db7535180d3c71689169a and Jev 1.13.0 on the same 3,880 records from 13 subsets. People wrote the labels. None of the tasks is one of the ten training categories. The instructions were fixed before either model ran, and both models saw the same wording. Nimble ran at temperature 1.0. The machine was an AMD Radeon RX 6800 XT with part of the model in system memory, so the guide does not treat the timing as a serving result. Each subset is one seeded draw.

The unweighted average across the 13 subsets is 74.8% for Nimble and 76.0% for Jev. Counted by record, the figures are 75.9% and 77.3%. Choice subsets average 81.6% against 82.9%. True/false subsets average 80.2% against 84.6%. The three score subsets average 54.6% against 50.1%. Nimble leads on that group.

Five subsets separate the models on an exact McNemar test at p below 0.05. Jev is ahead on Civil Comments, PAWS, German MASSIVE, and VitaminC dev. Nimble is ahead on SummEval relevance. The other eight do not separate them on this draw. The same 350 MASSIVE utterances, switched from English to German, take Nimble from 86.9% to 83.4%: right only in English on 18, only in German on 6, p=0.023. Jev moves from 87.4% to 86.9%, 11 against 9, p=0.82. SummEval relevance rests on 15 articles, and the consistency subset on 9. Civil Comments is mostly non-toxic: 6 records are right only for Nimble and 38 only for Jev. The temperature fit below was not applied on this suite. We did not rerun the subsets.

On September 21, Yufan Zhuang posted an attack on the published checkpoint. The attacker appends at most 40 words after a blank line, from 24 handwritten templates, then changes those words one at a time. His chart counts how many of 168 documents flip inside 300 queries. Published Nimble flips on 72% when the attacker sees only the decision, and on 97% when the attacker also sees the probabilities. Untuned Qwen3.5-9B flips on 95% and 100%. The same recipe retrained for 334 steps sits at 64% ± 28 and 90% ± 11. Twice the steps, 669, sits at 49% ± 25 and 88% ± 15. Training on the injected prompts is the row at 6% ± 3 for the decision alone and 24% ± 3 when probabilities are visible. The retrained rows are a mean and a standard deviation over two seeds. The chart’s caption says this is a fix only the vendor can make. We did not rerun the attack.

On September 22 they fitted a temperature of 2.179 for revision 93ec5d6 of the original adapter. Dividing the logits by that number leaves the winning answer unchanged: still 292 of 324 on the holdout, and 220 of 300 on a second set that is in neither the training file nor the holdout. At temperature 1 on that second set, the picked answer carried an average probability of 0.89 and was right 73% of the time. After the fit, expected calibration error on those 300 moves from 0.128 to 0.066, log loss from 0.692 to 0.555, and Brier score from 0.348 to 0.295. On the 324, expected calibration error moves from 0.052 to 0.054. On the 64 rating questions inside the 324 it rises from 0.105 to 0.177. The fit used the CUDA path that loads the adapter without merging it. The Mac and Linux quickstarts use merged weights, and the README says those logits were not checked again.

On September 24 the Hugging Face repository Bespoke-Nimble-9B moved to a newer checkpoint. The card sets the context limit at 8,192 tokens and allows up to 255 choices per field. The default temperature is 1.0, and the card says not to reuse 2.179. The earlier adapter remains at revision original-2676. The README says a separate v2 repository is unchanged, and that both scorers apply a transferred temperature of 2.179078721266035 to it. The Hub page for that v2 repository did not load. The card and the README print no 324 row and no public-suite row for the September 24 checkpoint. The 90.12% is the original release’s holdout. The 74.8% is revision 594dfdcf at temperature 1.0.

The current README says the scorer accepts only text, even though Qwen3.5-9B includes a vision component. On September 19 Sathiamoorthy posted one floor plan, with rooms marked Bedrm 1, Bedrm 3, and Bedrm 4, and wrote that Nimble counted the bedrooms and bathrooms while the base Qwen struggled. He did not score Jev, because Jev has no image input. The post is one example, and the repository does not publish a scored image set.

A try-it page documents a public Modal demo, model id nimble-latest, with no API key. The GPU sleeps after two minutes without requests. That page says the probabilities use the fitted temperature, and that choice and score fields allow 2 to 26 options. The September 19 note in the README raised the hosted API’s prompt cap to 8,192 tokens. Training used prompts up to 2,048. On September 23 Sathiamoorthy wrote that hosting Nimble was harder than training it, and that a better version was planned. The post names no ship date.

Benchmark Heaven posted this 9B on JevBench’s geometric mean at 63.5, with Intelligence 78.6 and Speed 82.5. The write-up of that bench is JevBench. Florian’s post the next day said 63.7 and rank 17. The figure this desk checked on that table is 63.5.

On September 25 Harold, posting on that account, gave Nimble 9B a later row: hard-tier accuracy 65.5%, JevBench Score 60.5, and about $0.166 per 1,000 decisions. He wrote that the 90.1% versus 93.2% figures are not from that ledger. We did not reopen the board.

On September 30 Ollama posted ollama pull nimble on a local /v1/systemone endpoint in version 0.35. The model card’s mean on the 13 public datasets is 75.7% for Nimble, and it lists Jev at 76.0% from Bespoke Labs’ published run. The blog, dated September 29, puts the Pac-Man example at 91 ms on an M5 Max. We did not run that build.

Opened October 6, Bespoke-Nimble-9B-v3 is a LoRA on Qwen3.5-9B. The card prints Decision Index 0.2.1 at 56.88 and says the license is CC BY-NC 4.0. The base model on the card is Apache 2.0. The card describes a choice among options, or a yes or no. Pull 29844 on ggml-org/llama.cpp merged October 2 and names that repository. Release v0.6.0, published October 5, lists nimble on /v1/systemone. The server header at that tag does not mark Nimble as an image model. 56.88 is the v3 card. It is not the 292 of 324 holdout, and it is not the 74.8% public suite. We did not run the server. The server write-up is llamacpp-jev.