Published
Updated
OpenBles Firewall 1 catches 119 of 120 agent attacks and flags no benign look-alike
This topic was created 21 days ago, and the information it contains may have evolved or changed since then.
OpenBles Firewall 1 caught 119 of 120 agent attacks and flagged none of 120 benign look-alikes on its held-out test. The README prints classic-set recall at 55 of 60, with 2 of 56 benign prompts flagged. The adapter is MIT, 31.8 MB, on fastino/GLiNER2.5-Decide. We did not run the weights.
On October 9, 2026 at 16:43 UTC, @stas_sorokin_ posted OpenBles Firewall 1. The attachment is a video of about 24 seconds. The model reads text an agent is about to read and picks injection or benign. The weights are a 31.8 MB adapter. The repo is stas4000/openbles-firewall. A reply the same hour links openbles/firewall-1. We did not open the card.
The README, opened October 11, says the code and the adapter are MIT. The adapter sits on fastino/GLiNER2.5-Decide, which is Apache 2.0. Training used the train splits of two public sets and took about 8 minutes, 360 steps. The intro says about 33 ms per text on a laptop. A later sentence says about 35 ms per text on an M4 Pro. The post says 33 ms. The repo was called jev-injection-guard until October 9.
The agent test split has 240 texts: 120 attacks across 12 families, and 120 benign look-alikes. At a probability of at least 0.5, OpenBles caught 119 of 120 and flagged none of the look-alikes. The README prints F1 0.996 and a false-flag rate of 0%. Jev on every text, jev-1.13.0 on October 2, prints F1 0.956 and a false-flag rate of 9.2%. The post says Jev caught 120 and flagged 11 clean texts. Nine point two percent of 120 is 11. Untuned GLiNER2.5-Decide matches Jev on that split, F1 0.956 and 9.2% false flags.
The classic split is different. It is the test split of deepset/prompt-injections, 116 texts, 60 labeled injection. OpenBles caught 55 of 60 and flagged 2 of 56 benign prompts. The README prints F1 0.940 and a false-flag rate of 3.6%. Jev on that split caught 33 of 60, flagged nothing benign, and prints F1 0.710. The README says much of what Jev lets through there is labeled injection and reads as an ordinary request.
| Set | OpenBles Firewall 1 | Jev 1.13.0 |
|---|---|---|
| Agent attacks, of 120 | 119 caught, 0 of 120 look-alikes flagged | 120 caught, false flags 9.2% |
| Classic injections, of 60 | 55 caught, 2 of 56 benign flagged | 33 caught, 0 false flags |
A third file, bench/sneaky-inputs, is 164 inputs. The README says the model caught 60 of 60 hidden HTML attacks and 12 of 12 poisoned tool results, at about 65 ms per input on a MacBook Pro M5. It also says clean full pages, 28 of 64, and clean skills, 8 of 10, still get flagged too often, because the training texts were short. That file is not the table above.
The README says train and test come from the same two datasets, so these scores are in distribution, and that text unlike them should be expected to score worse. One run. We did not run the weights. This checkpoint is not a row on the JevBench page.
Other open checkpoints are on decision models like Jev.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
@stas_sorokin_, October 9, 2026, 16:43 UTC, status 2108599327920632173. The attachment is a video of about 24 seconds. The text says 119 of 120 attacks caught, 0 clean texts flagged, 33 ms on a laptop, 31.8 MB, MIT. It says Jev caught 120 and flagged 11 clean texts. The reply links the GitHub repo and says the classic set is 55 of 60 caught, with 2 of 56 benign prompts flagged, one run on October 9. A later reply links https://huggingface.co/openbles/firewall-1. We did not open the Hugging Face card.
README https://github.com/stas4000/openbles-firewall, opened October 11. MIT for the code and the adapter. The adapter is 31.8 MB on fastino/GLiNER2.5-Decide, Apache-2.0. The intro says about 33 ms per text on a laptop. A later line says about 35 ms per text on an M4 Pro. Training is about 8 minutes, 360 steps.
Agent test split, 240 texts, 120 attacks and 120 benign look-alikes. OpenBles at p at least 0.5: F1 0.996, false flags 0%, 119 of 120 caught. Jev on every text: F1 0.956, false flags 9.2%. Untuned GLiNER2.5-Decide matches that Jev F1 and that 9.2%. Classic test, 116 texts, 60 labeled injection. OpenBles F1 0.940, 55 of 60 caught, 2 of 56 benign prompts flagged, false-flag rate 3.6%. Jev on that set is F1 0.710, 33 of 60, and 0 false flags. The README says the Jev rows were run on 2026-10-02 with jev-1.13.0. It also says train and test come from the same two datasets, so the scores are in distribution. We did not run the weights.
Compare
119 of 120, with 0 of 120 benign look-alikes flagged, is the agent test split. 55 of 60, with 2 of 56 benign prompts flagged, is the classic split. Jev's 9.2% on the agent split is 11 of 120, which is the post's "11 clean texts." Jev's classic row is 33 of 60 and 0 false flags. The 33 ms line is the post and the README intro. The 35 ms line is the M4 Pro sentence. Sneaky inputs, 60 of 60 hidden HTML attacks and 12 of 12 poisoned tool results, are a third file.
Terms
- OpenBles Firewall 1
- A 31.8 MB MIT adapter on fastino/GLiNER2.5-Decide. On the agent test split it caught 119 of 120 attacks and flagged none of 120 benign look-alikes.
- Classic set, OpenBles
- deepset/prompt-injections test split. The README prints 55 of 60 injections caught and 2 of 56 benign prompts flagged. Jev on that split is 33 of 60, with no false flags.