meraGPT posts Decider 1 ahead of Jev on a teacher-labelled set

Latent Node posted on September 22 that Decider 1 (sd-1) is available on meraGPT. The post says the typed-decisions benchmark scored it at accuracy 0.768 against Jev 1.13.0 at 0.727, with KL 0.096 against 1.442, at $0.03 per million input tokens and free output. The next post links the write-up: meragpt.com/blog/introducing-decider-1. The page is dated the same day and has no byline.

The blog describes three question types in one request: noul (probability of yes), choice (up to ten labels), and score (an ordered rubric). It says the state is read once, so extra questions about the same state cost about as much as one. A client that already uses the typesafe SDK is told to set TYPESAFE_BASE_URL to https://meragpt.com and use a meraGPT key.

The results table lists a prior that ignores the input (accuracy 0.470, KL 0.347, Brier 0.189), MiniLM-L6 fitted per question (0.587 / 0.262 / 0.143), ModernBERT-base fitted per question (0.646 / 0.223 / 0.119), TypeSafe Jev 1.13.0 (0.727 / 1.442 / 0.148), and Decider 1 (0.768 / 0.096 / 0.052). By type, Decider 1 leads on noul (0.840 versus 0.775), choice (0.733 versus 0.720), and score (0.739 versus 0.696). The fitted rows were trained on each workflow’s labels. The blog says Decider 1 and the Jev run were zero-shot, and that Decider 1 had not seen the four workflows or the twenty question schemas. The accuracy gap is 0.041. The KL gap is the larger number on the page.

The same page says how to read the score. Reference answers are the average of three samples from teacher models, so the number is agreement with that ensemble. Two single teacher samples agree 73.5% of the time. A model that predicts the average can sit above that rate. Every figure is from one evaluation run. No interval is printed.

Speed, from a client, network included: about half a second at the median for a request that answers every question in it. One serving worker is described at about 14,000 prompt tokens a second, about 80 decisions a second on these cases, before requests queue. A request may hold up to 4,096 tokens. A choice may hold at most ten labels; the blog says to split a longer list and add a “none of these” label. Price is $0.03 per million input tokens against Jev’s $0.042. Output is free on both.

The dataset linked from the blog is LocalLLaMA/typed-decisions. The viewer we opened shows four subsets (agent-trace observability, customer service, invoice processing, security incidents) at 400 rows each, and a combined view of 1,600 rows. The agent-trace subset is split 300 train and 100 test. Rows carry five questions, and the gold columns are probability distributions, which matches an averaged label. The blog’s scored count is 400 cases and 2,000 decisions. We did not download the files, so we cannot say which count the table used. We did not call Decider 1 or Jev.

JevBench is a different suite: 534 decisions and a composite, with Jev at 75.4 in the table this desk last read. Open-Jev scored 231 of those tasks and put Jev at 200/231. Security triage used 100 synthetic cases and one author’s labels, and Jev’s balanced accuracy there was 65.3%. None of those denominators is the typed-decisions table.