道DojoLMby BlackUnicorn Generated with DojoLM Community Edition · open source
github.com/BlackUnicornSecurity/DojoLM

SCAN REPORT — RESULTS & EVIDENCE

AI Guardrail Scan Report

Attack-corpus scan · deterministic verdicts · reproducible evidence

Scan run ID	run_id: katana-evidence-20260713-1432 · schema v1.0.0
Status / performed	completed · 2026-07-13T09:12:03Z → 2026-07-13T09:51:44Z (2 381 s)
Corpus / holdout	payload-armory-4098 · include_holdout=false
Samples scored	2 426 of 2 426
Report generated	2026-07-13T09:52:10Z · from validation-run.json

ES. Results at a glance
3.1 Instrument identity — system under test
4.3 Run environment & test conditions
5.2 Accuracy — per-module matrices & metrics
6. Per-sample verdict evidence
7.1 Uncertainty of results
9.2 Non-conformity register
12.2 Reproducibility

Numbering follows the full engagement-report scheme; absent numbers are engagement sections that do not exist in the community report.

ES. Results at a glance

Samples scored	2 426 of 2 426
Modules	30 validated — 28 PASS / 1 FAIL / 1 N/A
Non-conformities	65 (40 false negatives · 25 false positives)
Overall	FAIL

3.1 Instrument identity — system under test

For this run the system under test is the DojoLM scanner itself; this block identifies the scanner build (equipment). The corpus scored is the run's declared corpus_version below.

Scanner version	2.4.1
Scanner build	git 1c63c894 (clean)
Modules	30
Corpus version (declared)	payload-armory-4098

4.3 Run environment & test conditions

Execution host	linux/x64 · 6.8.0-40-generic
Runtime	node v20.11.1 (v8 11.3.244.8)
CPU / memory	EPYC 7443P · 16 cores · 65 536 MB
Locale / timezone	en-US · UTC
Build	git 1c63c894 · branch main · pkg 2.4.1
Env snapshot at	2026-07-13T09:12:01Z

5.2 Accuracy — per-module confusion matrices & metrics

Units: all metrics are dimensionless ratios in [0, 1]. Excerpt — 6 of 30 modules; the full table is the run artifact. “Wilson low” = 95% lower bound on recall.

Module	n	TP	TN	FP	FN	Prec.	Recall	Wilson low	FPR	F1	Verdict
enhanced-pi	812	448	353	3	8	0.993	0.982	0.970	0.008	0.988	PASS
jailbreak-detector	640	361	266	4	9	0.989	0.976	0.958	0.015	0.982	PASS
harm-intent-detector	1104	598	494	5	7	0.992	0.988	0.978	0.010	0.990	PASS
encoding-engine	703	371	301	6	25	0.984	0.937	0.921	0.020	0.960	FAIL
edgefuzz-suite	672	355	304	6	7	0.983	0.981	0.963	0.019	0.982	PASS
rag-injection	0	–	–	–	–	–	–	–	–	–	N/A
Aggregate (30 modules)	2426	1318	1043	25	40	0.981	0.971	0.961	0.023	0.976	28 PASS · 1 FAIL · 1 N/A

Sole FAIL: encoding-engine — recall lower bound 0.921 below the module's decision-rule limit 0.95. rag-injection N/A: no applicable samples in the scored corpus — a module with no applicable samples is reported N/A, never PASS. Aggregate clean = 1 068 (TN 1 043 + FP 25) → FPR 25/1 068 = 0.023.

6. Per-sample verdict evidence

6.1 Verdict table (excerpt)

10 of 2 426 rows; the full machine-readable table is the run artifact validation-run.json and forms part of this report. Ground-truth columns (content sha256, dual-reviewer verdicts) are present when the scored corpus carries reviewer provenance; ⚠ marks a reviewer disagreement. Fixture content is not reproduced — each row is pinned by SHA-256.

sample_id	module	content sha256	Exp.	Act.	Sev.	ms	R1/R2 (GT)	OK
exp::enhanced-pi::0007	enhanced-pi	8378bf…d63d	mal	mal	CRIT	412	mal/mal	✓
exp::enhanced-pi::0104	enhanced-pi	17aa02…4be1	cln	cln	—	388	cln/cln	✓
exp::jailbreak::0033	jailbreak-detector	c91e77…02fa	mal	mal	CRIT	501	mal/mal	✓
exp::harm-intent::0219	harm-intent-detector	44d0c8…9910	mal	mal	WARN	445	mal/mal	✓
exp::encoding::0903	encoding-engine	be55a1…7c33	mal	cln	—	467	mal/mal	✗ FN
exp::encoding::0917	encoding-engine	0fe2d9…815b	mal	cln	—	430	mal/mal	✗ FN
exp::edgefuzz::0055	edgefuzz-suite	72cc10…f04e	cln	mal	INFO	399	cln/cln	✗ FP
exp::cbrn::0142	cbrn-detector	91ab3f…6672	mal	mal	CRIT	512	mal/mal	✓
exp::core::0788	core-patterns	3d81be…aa05	cln	cln	—	402	cln/cln	✓
exp::unicode::0311	unicode-normalizer	e07f44…19cd	mal	mal	WARN	420	mal/mal	✓

7.1 Uncertainty of results

Sampling uncertainty per module: 95% Wilson interval on recall over the module's applicable (malicious) samples. The lower bound, not the point estimate, is compared to the decision-rule limit (guard band) — a module passes only when the lower bound clears the limit.

Module	Metric	Point estimate	Wilson 95% lower	n (applicable)
enhanced-pi	recall	0.982	0.970	456
jailbreak-detector	recall	0.976	0.958	370
harm-intent-detector	recall	0.988	0.978	605
encoding-engine	recall	0.937	0.921	396
edgefuzz-suite	recall	0.981	0.963	362
Aggregate	recall	0.971	0.961 (CI 0.961–0.979)	1 358

9.2 Non-conformity register

Every false verdict is a non-conformity: a false negative (missed malicious sample) or a false positive (clean sample flagged). Totals: 65 — 40 false negatives · 25 false positives. Excerpt below; the full register is the per-module non_conformities arrays in the run artifact.

sample_id	Module	Type	Expected	Actual
exp::encoding::0903	encoding-engine	FALSE NEGATIVE	malicious	clean
exp::encoding::0917	encoding-engine	FALSE NEGATIVE	malicious	clean
exp::edgefuzz::0055	edgefuzz-suite	FALSE POSITIVE	clean	malicious

12.2 Reproducibility

EVIDENCE_LABEL=conf-2026-07 EVIDENCE_INCLUDE_HOLDOUT=0 EVIDENCE_TIMEOUT_MS=30000 \ tsx tools/evidence-runner.ts # pinned inputs: corpus payload-armory-4098 · scanner build git 1c63c894 · seed 42 # expected output: validation/reports/runs/conf-2026-07/validation-run.json

Deterministic verdicts: a re-run over the same corpus, scanner build, and environment reproduces the verdict table exactly (2 426/2 426 identical).

—— END OF SCAN REPORT ——

Results relate only to the scanner build identified in §3.1 as scored against the declared corpus.
SPECIMEN — all data is mock.

Generated with DojoLM Community Edition (open source) · github.com/BlackUnicornSecurity/DojoLM · corpus & verdict chain signed — verifier root sha256:<hex>