meraGPT posts Decider 1 ahead of Jev on a teacher-labelled set

Latent Node posted on September 22 that Decider 1 (sd-1) is available on meraGPT. The post says the typed-decisions benchmark scored it at accuracy 0.768 against Jev 1.13.0 at 0.727, with KL 0.096 against 1.442, at $0.03 per million input tokens and free output. The next post links the write-up: meragpt.com/blog/introducing-decider-1. The page is dated the same day and has no byline.

The blog describes three question types in one request: noul (probability of yes), choice (up to ten labels), and score (an ordered rubric). It says the state is read once, so extra questions about the same state cost about as much as one. A client that already uses the typesafe SDK is told to set TYPESAFE_BASE_URL to https://meragpt.com and use a meraGPT key.

The results table lists a prior that ignores the input (accuracy 0.470, KL 0.347, Brier 0.189), MiniLM-L6 fitted per question (0.587 / 0.262 / 0.143), ModernBERT-base fitted per question (0.646 / 0.223 / 0.119), TypeSafe Jev 1.13.0 (0.727 / 1.442 / 0.148), and Decider 1 (0.768 / 0.096 / 0.052). By type, Decider 1 leads on noul (0.840 versus 0.775), choice (0.733 versus 0.720), and score (0.739 versus 0.696). The fitted rows were trained on each workflow’s labels. The blog says Decider 1 and the Jev run were zero-shot, and that Decider 1 had not seen the four workflows or the twenty question schemas. The accuracy gap is 0.041. The KL gap is the larger number on the page.

The same page says how to read the score. Reference answers are the average of three samples from teacher models, so the number is agreement with that ensemble. Two single teacher samples agree 73.5% of the time. A model that predicts the average can sit above that rate. Every figure is from one evaluation run. No interval is printed.

Speed, from a client, network included: about half a second at the median for a request that answers every question in it. One serving worker is described at about 14,000 prompt tokens a second, about 80 decisions a second on these cases, before requests queue. A request may hold up to 4,096 tokens. A choice may hold at most ten labels; the blog says to split a longer list and add a “none of these” label. Price is $0.03 per million input tokens against Jev’s $0.042. Output is free on both.

The dataset linked from the blog is LocalLLaMA/typed-decisions. The viewer we opened shows four subsets (agent-trace observability, customer service, invoice processing, security incidents) at 400 rows each, and a combined view of 1,600 rows. The agent-trace subset is split 300 train and 100 test. Rows carry five questions, and the gold columns are probability distributions, which matches an averaged label. The blog’s scored count is 400 cases and 2,000 decisions. We did not call Decider 1 or Jev. We did not recompute the table.

On September 30, 2026 the dataset card says the leaderboard is the test split: 400 cases, 2,000 decisions. The dataset server lists a combined config of 1,200 train rows and 400 test rows, and each of the four workflows at 300 train and 100 test. The viewer’s 1,600 rows are that combined config, train and test together. A dataset-level total of 3,200 counts the combined config a second time beside the four workflows.

The card’s note on the Jev 1.13.0 row says it was measured on September 18, 2026 through jev-latest, which reported jev-1.13.0, on all 2,000 decisions, with zero errors, at $0.016. The same table prints a p50 latency of 710 ms and an ECE of 0.144.

On September 23 the same account posted a second comparison. Decider 1 was run through a harness the posts call jev-eval, described as open source, with 300 labelled cases per task. The summary says accuracy stayed within 1 to 3 points of Jev, cost per call was 2.7 to 4.3 times lower, and the clock was about 0.7 seconds against 0.2 seconds. A later post says the model could run three of the harness tasks and was about 3.5 times slower at the median. The summary also says calibration was much better on 5-level scores. It does not print an ECE. One reply prints a single task: yes/no sentiment, Jev 0.970 at $0.021 per 1,000 calls, Decider 1 0.943 at $0.0078 per 1,000 calls, 3.5 times slower. The account says the multiples move a lot by task. The thread links a playground at meragpt.com/playground. It does not link a repository or a results file. The name jev-eval in these posts is not tied, in the thread, to onlyoneaman/jev-eval, which is Aman Kumar’s public case dump. The 0.768 table is the September 22 blog. The sentiment row is this later harness.

JevBench is a different suite: 534 decisions and a composite, with Jev at 75.4 in the table this desk last read. Open-Jev scored 231 of those tasks and put Jev at 200/231, and on September 23 added a 27B checkpoint at 197/231. Security triage used 100 synthetic cases and one author’s labels, and Jev’s balanced accuracy there was 65.3%. None of those denominators is the typed-decisions table, and none is the yes/no sentiment cell.