CLM-8B caches the action list, and a fine-tuned head scores 31 of 38 DeepSWE holdouts

Jacky Kwok posted CLM-8B on September 23 at 23:37 UTC. The repo is Contrastive-LM/CLM. The weights are Contrastive-LM/CLM-v0.1-8B. The README says the code and those weights are Apache 2.0. The citation names Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Marco Pavone, Christopher Ré, and Azalia Mirhoseini with him. The longer write-up is a Notion page.

CLM trains a state encoder and an action encoder with a bidirectional InfoNCE loss, so a state is pulled toward the action that was taken and pushed away from the others. At inference the backbone is a frozen Qwen3-8B with last-token pooling, and each projection head is 20 million parameters. A typed question is the state plus the option texts. A softmax over the dot products is the distribution. clm-serve answers POST /v1/systemone on port 8700. The encoder is a vLLM embeddings server on port 8090. Confidence on a Choice is the top probability minus the mean of the others.

The README’s recipe has three stages. Pre-training is about 60 million Nemotron DQA question-answer pairs, the question as the state and the answer as the action. Mid-training adds about 30 million synthetic hard negatives written by Gemini 2.5 Flash-Lite. Post-training is about 1 million agent steps from the Agent Data Protocol, Endless-Terminals, and LiteCoder-Terminal-SFT. Forty percent of that last mixture is DQA replay. With the replay, Nemotron hard-negative top-1 accuracy moves from 69% to 68.5%. The same number of agentic steps with no replay drops it to 56.2%. On about 100,000 held-out questions, one gold answer and ten hard negatives each, pre-training alone is 52.1% top-1. A short mid-training stage reaches 69.2%. Training on hard negatives from the start peaks at 62.4%. The README says the test InfoNCE loss falls as a power law in compute, dataset size, head size, and encoder size, and that encoder size moves it the most. At a fixed compute budget the preferred head size grows about linearly with tokens, roughly 310 tokens per parameter.

The zero-shot section says CLM-8B stays level with Jev on computer use, games, and tool calling, at up to 9 times lower latency. The file names WikiRacing, where the candidate list is long, and the T-Rex game, where the same actions come back. A thread post says that at about 1,000 candidates CLM is 13 times faster than Jev, because the action embeddings can be cached on their own. That post says Jev caches state only. The README’s own RTX 4090 table is a smaller loop: a new state stays near 28 ms for 3 actions or 50, and a revisited state drops from 1.7 ms to 0.6 ms once the vector cache is on. A loop that never repeats a state still pays for the encoder.

The verifier numbers are a different job. For each task the authors sample several solutions, Opus 5 on DeepSWE and Fable 5 on Terminal-Bench 2.1, and CLM or Jev picks one. The set is 38 held-out DeepSWE tasks and 30 held-out Terminal-Bench 2.1 tasks. Latency is on an H100. The reproduce comment in the README says 31/38 = 81.6% for DeepSWE. Terminal-Bench 2.1 is printed as 87.6%, and that line does not give the count as a fraction. The same paragraph says Jev scores below pass@1 on these tasks, and that the fine-tuned CLM head runs 4.1 to 5.7 times faster than Jev. The launch post said 4 to 6. The public DeepSWE agent board from May 26, 2026 is a different measurement: the agent writes the patch on the full task list, and that snapshot lists gpt-5.5 at 70%. CLM’s 81.6% is a fine-tuned verifier on 38 held-out tasks.

The thread says a multimodal CLM-35B is in training, with a release planned for early the next month. The reproduce command downloads Contrastive-LM/deepswe-clm-heads-8k and runs evaluation/bon_eval.py. We did not run it, and we did not call either model.

On September 26 Oleh Aleinyk published a Kev-versus-Jev rerun of fixtures from his earlier CLM report. The page says the CLM column is historical, not a new CLM run. On the same 1,253 function-selection requests, that column is 76.86% for name plus description, 58.42% with parameters in prose, and 47.49% on full JSON. Historical positive F1 on the narrow fixtures is 0.18 for tool-result relevance, 0.57 for subgoal completion, and 0.36 for file selection. Those figures are a different job from the DeepSWE 31/38 verifier. The fresh Jev and Kev counts are in the Kev story.