DiffusionGemma as Jev reads one denoising step, and a hand chart sums to 198 of 201

Google’s Gemma account posted on September 29 that vLLM can turn DiffusionGemma into a Jev-like model. The server seeds a canvas with a response template and, in one denoising step, reads confidence and a probability distribution for a yes/no question, a multiple-choice question, or a scored question. The next post thanks the vLLM project and Matt Mastracci and links pull request 57250. Lucas Wilkinson merged it on September 22.

The name is older than that post. On September 18 the same account called the setup DiffusionGemma as Jev, quoted Matt’s September 17 comparison, and put a single pass at about 0.2 seconds on a DGX Spark. That post says bidirectional attention yields well-calibrated decision distributions. Matt’s September 16 thread says the probabilities are the model’s own marginals, and that they are not calibrated against labelled data. He averages four reads with different noise and reports the standard error.

The model card lists 25.2 billion parameters, 3.8 billion active, a canvas of 256 tokens, a 262K vocabulary, and Apache 2.0. The specification table says text and image. The introduction also names video. Normal generation denoises a canvas of up to 48 steps. The Jev-style path asks for the first step only.

The pull request is the machinery. A discrete diffusion model denoises the whole canvas on each forward pass. If the canvas already holds the answer’s fixed text, and only the answer slots are noise, that one step is a distribution over each slot. The example server is not a standard vLLM route. It speaks POST /v1/systemone, with noul, choice, and score. A label has to be a single token, so a long option name is mapped to a short one. The write-up says that with 26 options only 1 to 8 of the label ids ranked in the top-k, which is why the sampler now asks for those ids.

On the pull request’s DGX Spark test, one-way traffic is 8.7 requests a second at 0.12 seconds. At 32-way it is 54.0 requests a second at 0.58 seconds, about 162 decisions a second. The setup is read-only single reads, a 32-row canvas, and three decisions per request. A programming-language battery in the same note is 10/10. Human language is 9/10: Portuguese was called French, at 0.47. Unit comparison is 10/12.

mmastrac/djev is Apache 2.0. The README says this repository is the home of that example server, and that the copy inside the pull request is a snapshot. The vLLM recipe, updated September 23, pins a nightly image, serves the model as dgemma, and starts structured_server.py on port 8011 with a canvas of 64. The curl example in that recipe sends "model": "jev-latest". The djev README says a router that discovers models from GET /v1/models has to send the served name, dgemma.

Matt’s September 17 chart is eight hand sets. The fractions on the bars add to 198 of 201 for DiffusionGemma and 191 of 201 for Jev. Both are 12/12 on tickets, 10/10 on code language, 10/10 on a 10-way language set, 10/10 on a 26-way language set, 10/12 on unit comparison, and 38/38 on entity type. Reviews are 20/20 against 19/20. PII per word, five sentences, is 88/89 against 82/89. The entity-type row is labelled “42 words” and scored 38/38. He called the pair roughly tied, and said DiffusionGemma wins a bit on the PII test. The disagreement table shows the unit-comparison misses landing on different items: each model is wrong on two of the twelve, and they are not the same two. The items, the code, and the per-item outputs are not in the thread.

The latency chart on that thread times a warm second pass on the DGX Spark, one read per request, against Jev on one kept-alive connection. It subtracts a measured 46 ms round trip from Jev and says that trip is still inside the request. One read is faster on seven of the eight sets. Entity type is the exception, 215 ms against 166 ms. With the default policy that re-reads an uncertain question up to four times, DiffusionGemma is slower than Jev on all eight. On September 25 he posted that a lightly tuned Vertex A100 80 GB endpoint, about $3,000 a month, runs 250 decisions a second at a latency on par with the official API. That post has no table.

Chirag’s sieve README, MIT, says reversing the options flipped 24% of DiffusionGemma-Jev’s answers. Averaging three reads in shuffled order halved that, and added 17 to 26 points on sets with 60 to 151 options. The chart on the September 26 post prints 23.6% on one read, 14.3% on two, and 11.8% on three, against GLiNER at 3.0% on one read. The point gains on that chart are +17 on CLINC150 (151 options), +26 on MASSIVE Hindi (60), +24 on MASSIVE English (60), and +18 on Banking77 (77).

Josh Brown posted a Perch Bug Bench chart the same day as Google’s vLLM note. It is average precision on 2,220 public before-and-after pairs: DiffusionGemma Jev 46.8%, Jev 46.2%, SemIf 44.6%, Span-01 43.8%, CLM-v0.1-8B 42.1%. His parent post says the rule-based checks gave the strongest results, so 46.8% is the standard-detection figure on the chart. The source line points at perchscan.com/benchmark-results. That URL returned 404 when we opened it.

A separate replay, Hangzhi/diffusion-jev-sglang, measured September 21 on one A100 80 GB. On the 231 public JevBench tasks at commit c6004e0, local DiffusionGemma is 196/231 (84.8%) against the published Jev 1.13.0 artifact of 200/231 (86.6%). Easy is 48/48 for both. Original is 70/72 against 71/72. Hard is 78/111 against 81/111. The client-observed median is 183.3 ms against the published 665.0 ms. The report says the local clock is loopback on the A100 and the Jev clock includes the production path from Germany, and that no fresh hosted run completed because the gateway worker exited before any hosted item was measured. Ten-bin expected calibration error on the text tasks is 0.1222. Twenty of 35 wrong text answers had a top probability of at least 90%. A 100-image flower test is 98/100, both misses sunflowers called daisies, with no hosted Jev image column. That service reads the last active denoising step. Text requests used 2 to 6 steps, and 212 of 231 used two, beside the one-step server in the vLLM example.

ywchiu’s summary is 100 decision points, five repeats, teacher-forced, on an 11-question design. djev-spark, named there as DiffusionGemma 26B-A4B, scores 32.2% on the four-field joint against Jev 1.13.0 at 61.4%. The six-field joint is 13.4% against 48.8%. Median latency is 1,230 ms against 749 ms. The file says djev varies from vLLM nondeterminism at a fixed seed, and that Jev is the system that varies on its own.

Maisa’s hosted djev is a different service, and djev-run is the Cloud Run recipe. The hosted JevBench row, and the 35 to 60 ms samples=1 note, stay on those pages. We did not serve the model.