Published
Updated
spark-s1-4b-v6 swaps in Qwen3.5 and prints a public-tier 83.1
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Abhishek Rai posted spark-s1-4b-v6, an Apache-2.0 LoRA on Qwen3.5-4B. The README's public JevBench proxy is Intelligence 83.1, hard 0.595 of 111. On the v1.5 page opened September 29 the same checkpoint is official rank 7 at 68.2, capability 65.9. The v1.5.4 page opened September 30 keeps 68.2 and lists official rank 13. The proxy is not that composite.
Abhishek Rai posted spark-s1-v6 on September 22. The post says the new checkpoint is stronger than v5, by up to 13.8% on the public JevBench hard tier, and that an NVFP4 copy keeps the reported BF16 accuracy. The repository is abhishek085/open-spark-jev, Apache-2.0. The model is a LoRA on Qwen3.5-4B, a hybrid of 24 Gated DeltaNet layers and 8 full-attention layers. The page says it is not Jev and not affiliated with TypeSafe.
The public-tier table, 231 of 534 JevBench v1.2 items: easy (48) is 1.000 for both the bf16 checkpoint and the NVFP4 copy. Standard (72) is 1.000 and 0.972. Hard (111) is 0.595 and 0.622. The README calls the summary a public-proxy Intelligence, 83.1 and 83.2. It says the judge tier is private, so this is not the official score, which also needs Calibration, Speed, and Cost measured under JevBench’s own protocol. Changelog 0.3.0, the same day, says own-split accuracy rose 6.2 and 5.0 points over v5-4b, and that Intelligence went from 74.2 to 83.1. The changelog does not print v5’s hard cell, so the post’s 13.8% is not a figure we could recompute from those two pages. Own held-out accuracy is listed at 0.929. The NVFP4 own-split number is marked as not remeasured. Serving p50 through vLLM is 74.9 ms in bf16 and 53.3 ms in NVFP4, which the page calls 1.40 times faster.
On Kev’s transfer-v4 suite, 764 items in each partition, the README says neither model trained on those sources. spark-s1-4b-v6 is 0.787 on the locked test. The page’s published Kev figures there are Kev-4B 0.806 and Kev-8B 0.780. On the development partition the same model is 0.756, against Kev-4B 0.790, Kev-8B 0.796, and Jev 0.857. The Kev and Jev cells are the ones Kev published. The spark cells are this repository’s measurements.
A 60-case tool-call set is listed as a development diagnostic: spark 0.900 at a p50 of 74.9 ms and 13.3 decisions a second, against a recorded hosted Jev reference at 0.917, 421.6 ms, and 2.3 a second. The page says the set is small, about five points of noise, and that the Jev timing includes the network. The changelog says two of five external sets slipped against the previous release: prompt-injection without context by 3.6 points, and this tool-call set by 1.7.
Training used 11,792 rows across 49 task packs, reshuffled into 19,568 examples with the option order randomized. The README says the released model used only those os-datagen rows, and that the full mix is not yet its own dataset release. An earlier 967-row subset is on Hugging Face. RLCD, tried twice on earlier releases, made results worse and is not in this checkpoint. The encode-once cache does not apply to this backbone yet, so each extra question is a full forward pass. The tool gate’s default is strict: allow needs probability 0.995, and anything lower asks a person. We did not run the model.
JevBench v1.5.0, opened September 29, lists spark-s1-4b-v6 at official rank 7, score 68.2. The axis table prints Intelligence 62.1, Calibration 69.7, Speed 85.5, and Cost 60.4, about $0.021 per 1,000 decisions, p50 0.41 s. The capability view prints 65.9 and capability rank 13. The 83.1 above is the README’s public proxy, a different number. We did not rerun v1.5.
The page we opened September 30 is v1.5.4. The same 68.2 is official rank 13. SemIf, still 68.7, is rank 12 on that page, and metask at 67.5 is rank 14. We did not rerun v1.5.4.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Primary post is @abhi43210, September 22, 2026, 19:58 UTC. It says v6 is stronger than v5, up to +13.8% on the public JevBench hard tier, and that NVFP4 matches BF16 accuracy. The README we read is Apache-2.0.
spark-s1-4b-v6 is a LoRA on Qwen3.5-4B (24 Gated DeltaNet layers and 8 full-attention layers). Own held-out accuracy 0.929. Public-tier table: easy 48 items 1.000 for both bf16 and NVFP4; standard 72 items 1.000 and 0.972; hard 111 items 0.595 and 0.622.
Public-proxy Intelligence 83.1 and 83.2. vLLM p50 74.9 ms bf16 and 53.3 ms NVFP4, called 1.40x. The README says only 231 of 534 v1.2 items are public, the judge tier is private, and this is not the official score, which also needs Calibration, Speed, and Cost under JevBench's protocol.
Changelog 0.3.0, dated 2026-09-22, says own-split accuracy rose 6.2 and 5.0 points over v5-4b and Intelligence went from 74.2 to 83.1. It does not print a hard-tier cell for v5. NVFP4 own-split accuracy is not remeasured.
Kev transfer-v4, 764 items per partition, measured by this repo: locked test 0.787, against published Kev-4B 0.806 and Kev-8B 0.780. Development partition 0.756, against Kev-4B 0.790, Kev-8B 0.796, and Jev 0.857.
Diagnostic 60 tool calls: spark 0.900 at p50 74.9 ms and 13.3 decisions per second; a recorded hosted Jev reference 0.917 at 421.6 ms and 2.3 per second. The README calls that set a development diagnostic, about plus or minus 5 points, and says the Jev clock includes the network.
Training data: 11,792 code or LLM-labelled rows, 49 packs, reshuffled to 19,568 examples. The full mix is not packaged as its own dataset release. RLCD was tried twice on earlier releases, made results worse, and is not in this checkpoint.
The state-cache path does not apply to this backbone yet. Changelog findings: ext-injection-noctx -3.6 and ext-toolcall-risk -1.7 versus the prior release. Default tool gate: allow needs 0.995, otherwise ask.
We did not run the model.
The page we opened September 30 is v1.5.4. The same 68.2 is official rank 13 there.
Compare
rev's September 22 table, a different harness, puts Qwen3.8-27B at hard 0.784 and Jev at 0.730 on 111 public hard items. Open-Jev's 9B is 66/111 on its hard slice, and Jev there is 81/111. spark's 0.595 is another author's public-tier run.
None of those is JevBench's 220-item hard tier, where Jev is 74.1% and Luna is 94.5%.
On v1.5.4 the same 68.2 is official rank 13. The September 29 page listed it at rank 7.
Terms
- spark-s1-4b-v6
- Abhishek Rai's Apache-2.0 LoRA on Qwen3.5-4B. It restricts the next-token logits to option labels. The README's public-proxy Intelligence is 83.1, which is not the official JevBench composite.
- public-proxy Intelligence
- The README's name for a score on the 231 public JevBench v1.2 items. The judge tier is private, and Calibration, Speed, and Cost are not in this number.