Published
Updated
ToddlerJev prints 68.70% on MMAU-mini, with weights under the Qwen Research License
ToddlerJev, posted October 18, 2026, prints 68.70% on MMAU-mini, 61.30% on MMSU, and 56.90% on MMAR. Weights are TeenageWorks/Toddler-Jev under the Qwen Research License. This desk did not run them.
Vyom posted ToddlerJev on October 18, 2026, at 22:19 UTC. The text asks what if Jev could hear, names an open-weight native-audio decision model, and says it can identify instruments, coughing, crying, and sobbing. The still on that post is the image saved for this page. The tweet’s Hugging Face URL used ToddlerJ-Jev. The card we opened is TeenageWorks/Toddler-Jev.
The blog, opened October 19, says the model takes audio, a textual question, and a candidate set, and returns a probability for each candidate in one forward pass. It does not transcribe first. Rank-16 LoRA on the Thinker of Qwen2.5-Omni-3B plus a pointer head, 8,421,888 trainable parameters. Audio and vision encoders stay frozen.
Native-audio scores on that page: MMAU-mini 68.70, MMSU 61.30, MMAR 56.90. The table reprints Qwen-Audio-Jev at 78.70, 84.06, 69.90 and AudioJev at 70.90, 61.70, 55.33. The blog says those comparator values are published results, not reruns, and that protocols, prompts, subsets, and scoring differ. ToddlerJev’s own counts are 1,000 MMAU-mini items, all 5,000 MMSU items, and a 1,000-item short-clip MMAR subset.
A100 40 GB, BF16: streaming update p50 111.9 ms on an 8-second window with 1-second steps, 30-second recording p50 123.5 ms. A paired 200-item MMAU diagnostic scores 73.50% with full audio, 67.00% with the first eight seconds, and 38.50% with silence. Reordering identical options changes the selected answer on 9.70% of MMAU-mini cases and 12.80% of MMAR. 15-bin expected calibration error 0.085 on MMAU-mini and 0.104 on MMAR.
Spoken-text suites on the blog use TeenageWorks-synthesized audio for ToddlerJev and original text for the comparison systems. Banking77 81.50 against a printed Jev 1.13.0 cell of 81.30. ScienceQA 84.00 against 95.70. MMLU 36.50 against 80.60. The blog says 8,147 cases across 15 suites, 8,145 predictions, two explicit failures, and that no competitor API was rerun.
The card, opened October 19, returned HTTP 200. Version 1.0.0. License field Qwen Research License, research and evaluation use. Commercial use needs the permissions in that file. The example JSON marks calibrated false. We did not run the weights.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Vyom, @vyomnpatel, October 18, 2026, 22:19 UTC, status 2111945342417854854. The text says ToddlerJev is an open-weight native-audio decision model, and that it can identify instruments, coughing, crying, and sobbing. It links the blog and huggingface.co/TeenageWorks/ToddlerJ-Jev. The card we opened is TeenageWorks/Toddler-Jev. The share image on this page is the still from that post.
Blog teenage.works/blog/toddler-jev, opened October 19. MMAU-mini 68.70, MMSU 61.30, MMAR 56.90. The table reprints Qwen-Audio-Jev at 78.70, 84.06, 69.90 and AudioJev at 70.90, 61.70, 55.33, and says those comparator values are published results, not reruns. ToddlerJev uses 1,000 MMAU-mini items, all 5,000 MMSU items, and a 1,000-item short-clip MMAR subset. Streaming p50 111.9 ms, 30-second recording p50 123.5 ms, one A100 40 GB, BF16. A 200-item MMAU diagnostic: 73.50% with full audio, 67.00% with the first eight seconds, 38.50% with silence. Candidate reorder changes the selected answer on 9.70% of MMAU-mini and 12.80% of MMAR. 15-bin ECE 0.085 and 0.104. Spoken Banking77 81.50 against a printed Jev 1.13.0 cell of 81.30. Those spoken suites use synthesized audio for ToddlerJev and original text for the comparison systems.
Hugging Face card TeenageWorks/Toddler-Jev, opened October 19, HTTP 200. Version 1.0.0. Base Qwen2.5-Omni-3B, rank-16 LoRA, 8,421,888 trainable parameters. License field Qwen Research License, research and evaluation use, commercial use needs the permissions in that file. predict() returns a candidate and a probability for each option. calibrated false in the example JSON. We did not run the weights.
Compare
68.70 is ToddlerJev's MMAU-mini cell on 1,000 items. Qwen-Audio-Jev's 78.70 is the paper's cell, reprinted here as a published number, not a rerun. AudioJev's 70.90 on that same ToddlerJev table is also a reprint. Do not merge the three.
81.50 on spoken Banking77 is ToddlerJev on synthesized audio. The Jev 1.13.0 cell 81.30 on that table is pinned public predictions on text. The blog says no competitor API was rerun.
111.9 ms is an A100 40 GB streaming update, 8-second window, CUDA synchronized, excluding file decode and network. It is not TypeSafe Jev's hosted clock.
The Qwen Research License is not Apache-2.0. basal-omni-1.0-ultra is Apache-2.0 on Gemma 4, text and images, no audio.
Terms
- ToddlerJev
- Teenage Works' open-weight audio decision model, Hugging Face TeenageWorks/Toddler-Jev, Qwen Research License. Posted October 18, 2026. MMAU-mini 68.70%. Base Qwen2.5-Omni-3B with a rank-16 LoRA and a pointer head.
- Qwen Research License
- The card's license for this derivative of Qwen2.5-Omni-3B. It permits research and evaluation use. Commercial use needs the permissions in that file.
- silence gap
- On a paired 200-item MMAU diagnostic, ToddlerJev scores 73.50% with full audio and 38.50% when the signal is replaced by silence.