Published
Updated
Jevrail scores a shell command before it runs, and the printed corpus leaves the model off
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Saman Pandey posted jevrail, an MIT hook that asks Jev whether a shell command is likely to destroy something, then blocks, asks, or allows it. The published eval is an 80-line corpus with the model switched off: 22 of 22 catastrophic commands are caught, and 15 of 26 safe commands still raise a false ask.
Saman Pandey posted jevrail on September 25. It sits in front of a coding agent’s shell. The hook reads the command, asks how likely it is to destroy something, and then denies, asks, or allows it. The repository is MIT. The demo page is jevrail.samanp.xyz.
The sample config pins jev-1.13.0 and https://api.typesafe.ai. fail_mode defaults to ask. The HTTP timeout is 1,500 ms, and doctor warns outside 500 to 5,000. TYPESAFE_API_KEY overrides the key stored in ~/.config/jevrail/config.json. no_model keeps the decision on local rules and sends nothing out. The sample bands put destroys_uncommitted at ask 0.35 and deny 0.80, and touches_production at ask 0.30 and deny 0.75. The README says the model can only tighten a decision that the hard rules already made.
jevrail install --agent claude merges a PreToolUse hook into Claude Code’s settings. The opencode install writes a plugin. The README says Codex’s schema is unverified, and that opencode subagents may skip the hook. Coverage in the MVP is Bash. Write, Edit, and MCP tools are outside it.
The explain box for git reset --hard HEAD~3 prints 0.91 on uncommitted work and a deny. The README calls that output illustrative, from a run with a real key, and says the probabilities move with the model version and the repo. The evaluation file is the place with a count.
docs/BENCHMARK.md records one local run, jevrail eval on an 80-entry corpus with --no-model. Catastrophic recall is 22 of 22. Twelve of those are tier-0 hard denies. The other ten are the degraded ask. False asks on safe commands are 15 of 26, 57.7 percent. Tier-0 recall alone is 12 of 22, 54.5 percent. The corpus is hand-written, plus obfuscated commands and a short sample file. The note says the defaults stay unproven until someone tunes them on more than 300 entries. That file does not print a run with Jev turned on. We did not run the eval.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The September 25 post links github.com/SamanPandey-in/jevrail and jevrail.samanp.xyz. The repository is MIT. The sample config pins model jev-1.13.0, base_url https://api.typesafe.ai, fail_mode ask, and timeout_ms 1500.
Example bands: destroys_uncommitted ask 0.35 and deny 0.80, touches_production ask 0.30 and deny 0.75. The explain transcript in the README is labelled illustrative. The evaluation note's latest local run is jevrail eval on testdata/corpus/corpus.jsonl with --no-model, described as an 80-entry corpus.
Catastrophic recall is 100 percent, 22 of 22, of which 12 are tier-0 hard denies and the rest are a degraded ask. False asks on safe commands are 57.7 percent, 15 of 26. Tier-0-only recall is 54.5 percent, 12 of 22.
The note says defaults stay unproven until the corpus passes 300 entries. It does not print a run with the model on. We did not run the eval.
Compare
spark-s1's tool gate allows a command only at probability 0.995. semgate can block an HTTP request. This hook sits on Bash, Claude Code, and opencode, and the only scored file we read is the offline corpus.
The 57.7 percent false-ask rate is the no-model path. A live Jev column is not in that note. The README's 0.91 on git reset --hard is an example, not a row in the corpus.
Terms
- 0.80
- The sample deny band for destroys_uncommitted. At or above that probability the hook denies. The ask band on the same harm starts at 0.35. The evaluation note says these cuts are still starting guesses.
- tier 0
- Hard denies in the offline eval that do not need a model. 12 of 22 catastrophic commands in the 80-entry corpus were caught there. The other 10 were caught by the degraded ask path.