Strands Decider 2B scores 167 of 231 public JevBench tasks

Strands Labs posted Strands Decider 2B on October 1, 2026.

The post calls it a small open-source decision model for local experiments, and it links strands-labs/strands-decider and the Hugging Face org StrandsAgents.

What the model is

The README says the first model in the family has 1.9 billion parameters. The torso is Qwen3.5-2B-Base. The language-model head is removed, so the checkpoint does not generate text. A pointer head of about a million parameters scores each option from one forward pass.

The torso is adapted with a rank-16 LoRA, and the head runs in fp32. The blog says this is the second architecture. The first used a slot head, and the authors say that one did worse. The reference checkpoint is v19.

The CLI is pip install strands-decider. The documented ask command loads StrandsAgents/strands-decider-2B-hobson-v19 and accepts Choice, Noul, and Score questions. strands-decider serve exposes POST /v1/systemone. An example in the repository gates a weather tool inside a Strands agent.

The questions, the threshold, and the policy in that example were picked by hand.

Training is described as about 11 hours on one RTX 3090, or 1 hour 10 minutes on eight H100s with FAST=1. The README says the data sources and their licences are listed in the repository. We did not retrain it.

What v19 measures

The README’s performance table, and evaluation/README.md, print the same headline. On JevBench’s 231 public tasks, at the 3072-token window the run was preregistered at, v19 scores 167 of 231, 0.723. Brier is 0.342.

Expected calibration error is 0.052. On this repository’s split, easy, standard, and hard are 1.000, 0.875, and 0.505. Easy is 48 tasks, standard 72, hard 111.

Latency in that table is a median 115 ms and a 95th percentile of 299 ms per JevBench question on an RTX 3090 under WSL2. On an M3 Pro the warm median is 153 ms for tasks under 300 tokens, and 234 ms across all tasks. Served at the 4096-token default, the same notes print 168 of 231.

The authors call that one-task move noise. v20 reached 169 of 231 at 4096, missed four predictions, and did not replace v19.

Six retrains of the v17 recipe had a standard deviation of 3.2 tasks. The file says a single-run gap under about 10 tasks is unresolved. 167 against a published Jev count near 200 is larger than that band. This repository does not print Jev’s overall public accuracy.

It does print Jev at 0.27 on temporal_numeric, where v19 is also 0.267.

Where the rank is dated

evaluation/jevbench.md dates the ranking to the v1.4.2 board of September 25, 2026, 89 systems, using public-task accuracy. On that snapshot v19 is 50th overall and 3rd of 33 in the 2B class.

The blog repeats the 2B line and adds that, excluding models it calls just over 2B, the same snapshot is 1st of 30. The file is explicit that this is not the JevBench Score composite. Speed and cost were measured locally, and the sealed items are absent.

On that 2B list, decision-2b is 0.753, system-one-open is 0.732, v19 is 0.723, and the board entry for decider-2b is 0.710, 164 of 231. decider-2b uses the same Qwen3.5-2B-Base torso. The file says that board entry predates a v11 release which scores 175 of 231 on their harness, eight tasks ahead of v19’s 167.

Of the 231 tasks, v19 is right on 19 that the board-entry decider-2b misses, and wrong on 16 that the other gets.

The blog’s opening says the model can answer in tens of milliseconds. The latency section of the same post prints a median of around 115 ms on an RTX 3090. The graph of latency against token count is captioned as v18.

One limitation is in the evaluation README. With the state and the options held fixed, v19 gives the same answer to a changed question about 94% of the time. A question that reverses an obvious reading is often answered as if the reversal were not there.

On JevBench, 53 answers at confidence 0.9 or above were all right, at a mean claim of 0.957. The same note says the temperatures were fit on held-out classification, and that a threshold should be measured on the traffic it will see.