Published
Updated
On 492 tool calls, OpenRouter Armor's default Jev 1.13 scored 90.7%, 446 of 492
This topic was created 11 days ago, and the information it contains may have evolved or changed since then.
On 492 tool calls, OpenRouter Armor's default Jev 1.13 scored 90.7%, 446 of 492, at $0.0021 a call. On October 16, 2026, at 01:18 UTC, this desk's jev-1.13.0 noul was 0.98 on git push origin main and 0.01 on ls ./reports.
OpenRouter posted Armor on October 15, 2026, at 21:00 UTC. The launch tweet says the beta sits between the model and the application, scores each tool call with TypeSafe’s Jev, and beats Claude Code’s auto classifier at a fraction of the cost. The attachment is a video. The still on this page is the 492-call chart from the same thread.
That chart, status 2110838266920591515, is titled Accuracy vs Coding Test Set. Agreement with labels on 492 tool calls, a synthetic set seeded from real harness logs.
| Model | Agreement | Count | Printed cost |
|---|---|---|---|
| Jev 1.13, Armor default | 90.7% | 446 of 492 | $0.0021 / call |
| Clef | 88.4% | 435 of 492 | $0.0065 / call |
| Sonnet 5 (Claude Code) | 88.2% | 434 of 492 | ~$0.022 / call est. |
| GPT-6 Luna decisions | 82.7% | 407 of 492 | $0.0061 / call |
The footer says Sonnet 5 is a token estimate at 99% cached input. The bars mark agrees, false positives that over-block, and false negatives that miss an unsafe call. We did not rerun the 492.
The docs, opened October 16, 2026 UTC, mark Beta since October 2026. OpenRouter holds the tool call, sends the call, the rules, and the conversation to the decision model, and reads a probability per rule. The combined verdict is flagged or not_flagged. Hard-deny blocks and nothing clears it. Soft-deny blocks unless an allow rule matches or the user asked for that operation. Allow never blocks on its own. Default thresholds are 0.6 for deny and 0.7 for allow, set on Claude Code auto-mode test sets.
A custom agent adds { id: "armor" } to plugins. Claude Code auto mode through Ori can skip that entry. The launch tweet prints ori claude --model moonshotai/kimi-k3 -- --permission-mode auto. The docs page we opened prints kimi-k2-0905 in that same command. Docs print a median 279 ms of added latency and 334 ms at the 90th percentile on their Claude Code compatible rule set. During the beta, evaluation is not billed. Pricing will be announced before general availability. This desk did not send an Armor plugin request. There is no OpenRouter key in this environment.
Alex Atallah, 23:42 UTC, called it an auto permissions mode that works with any model. jacky, 21:12 UTC, wrote that the team keeps shipping production uses of Jev. Diogo Almeida quoted the launch at 21:41 UTC.
On October 16, 2026, Date headers 01:18:32 GMT and 01:18:33 GMT, this desk sent two POSTs to https://api.typesafe.ai/v1/systemone, model jev-latest. Both responses named jev-1.13.0 and returned HTTP 200. The question was a noul: does this command push git history to a remote. git push origin main scored 0.98, 330 input tokens, 21 output, desk clock 646 ms. ls ./reports scored 0.01, 328 input tokens, 21 output, desk clock 435 ms. GET /v1/models with the same key listed jev-latest and jev-preview. 658 input tokens at $0.042 per million, output uncharged, is $0.000028. These two calls are not the 492-row set, and they did not go through Armor.
446 of 492 is OpenRouter’s chart. TrueForge 37 of 40 is a different product. JevEval 0.254 is a faithfulness mean. Do not put those counts on one row. Armor is a plugin in front of Jev. It is not a catalog decision-model row, and it is not a door on Access.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Primary post is @OpenRouter, October 15, 2026, 21:00 UTC, status 2110838205654470840. Armor is in beta. It uses TypeSafe's Jev to score tool calls against policy before the agent runs them, and the tweet says that is better than Claude Code's auto classifier at a fraction of the cost. The attachment is a video of 28,800 ms. The share image is the 492-call chart from status 2110838266920591515, 21:00 UTC.
That chart title is Accuracy vs Coding Test Set. Agreement with labels on 492 tool calls, synthetic test set seeded from real harness logs. Jev 1.13, Armor default, 90.7%, 446 of 492, $0.0021 a call. Clef 88.4%, 435 of 492, $0.0065 a call. Sonnet 5 (Claude Code) 88.2%, 434 of 492, about $0.022 a call estimated. GPT-6 Luna decisions 82.7%, 407 of 492, $0.0061 a call. The footer says Sonnet 5 is a token estimate at 99% cached input. Legend: agrees, FP over-blocks, FN misses unsafe. We did not rerun the 492.
Docs https://openrouter.ai/docs/guides/features/plugins/armor, opened October 16. Badge Beta since October 2026. For each tool call, OpenRouter sends the call, the rules, and the conversation to the decision model and gets a probability per rule. Verdicts are flagged or not_flagged. Hard-deny, soft-deny, and allow. Default thresholds 0.6 for deny and 0.7 for allow, set on Claude Code auto-mode test sets. Median added latency 279 ms, p90 334 ms, on their Claude Code compatible rule set. During the beta, evaluation is not billed. This desk did not send an Armor plugin request. There is no OPENROUTER_API_KEY in this environment.
Alex Atallah, 23:42 UTC, status 2110879067855884587, calls it an auto permissions mode, more accurate, and powered by decision models like Jev. jacky, 21:12 UTC, status 2110841343819825489, says every week the team cooks production uses of Jev. Diogo Almeida, 21:41 UTC, status 2110848474866659826, quoted the launch with SO FKIN COOL.
This desk, Date headers 01:18:32 GMT and 01:18:33 GMT October 16, two POSTs to https://api.typesafe.ai/v1/systemone, model jev-latest, response jev-1.13.0, HTTP 200. Question, noul: does this command push git history to a remote. git push origin main scored 0.98, 330 input tokens, 21 output, desk clock 646 ms. ls ./reports scored 0.01, 328 input tokens, 21 output, desk clock 435 ms. GET /v1/models with the same key listed jev-latest and jev-preview. List price on 658 input tokens is $0.000028 at $0.042 per million, output uncharged. These two calls are not the 492-row set.
Compare
446 of 492 is OpenRouter's internal set. Clef 435 of 492, Sonnet 5 434 of 492, and GPT-6 Luna decisions 407 of 492 sit on that same chart. TrueForge's 37 of 40 is a different product and a different count. JevEval's 0.254 is a faithfulness mean. This desk's 0.98 and 0.01 are two noul calls on TypeSafe, not Armor's plugin path.
$0.0021 a call is the chart's Jev 1.13 cell. Sonnet 5 at about $0.022 is marked as a 99% cached-input estimate. Docs say Armor evaluation is unbilled during the beta. Those three cost lines are not one invoice.
279 ms median is OpenRouter's added latency on their Claude Code rule set. This desk's 646 ms and 435 ms are TypeSafe round trips on two short states. One call is not a latency ranking.
Terms
- OpenRouter Armor
- A beta plugin, posted October 15, 2026, that scores each tool call against prose rules with a decision model before the call runs. The default on the 492-call chart is Jev 1.13 at 90.7%, 446 of 492.
- hard-deny
- An Armor rule that blocks a matching call. An allow rule and user consent do not clear it. Default match threshold 0.6.
- 446 of 492
- Jev 1.13 Armor-default agreements on OpenRouter's synthetic coding set seeded from harness logs. Clef is 435 of 492. Sonnet 5 is 434 of 492. GPT-6 Luna decisions is 407 of 492.