Loading PHANTOM workspace…
We publish methodology, confidence intervals, and our integrity failures — because a security vendor’s numbers should survive the same scrutiny we apply to your code.
27 vulnerable targets, 9 safe controls. FP = 0 achieved through CRUCIBLE sandboxed exploit reproduction gate.
Ground truth established against public benchmark repositories and vulnerable targets:
Track 2 numbers evaluate the full end-to-end API pipeline running through the canonical `runScanSession` entrypoint and `PipelineOrchestrator.executeScan()`. It measures live service coordination latency, token budget ceilings, Redis Pub/Sub event delivery, and durable report object storage. Track 2 results are strictly separated from Track 1 offline component baselines.
During an earlier benchmark development cycle, an internal evaluation run was found to contain synthetic finding injection in test fixtures. While synthetic data is common in mock testing, claiming real-world benchmark precision against injected fixtures violates the core doctrine of offensive engineering.
The incident was caught by our verification gate, publicly documented in the release manifest, and fully remediated. All prior synthetic benchmarks were permanently downgraded to ARTIFACT_ASSERTED_ONLY.
To prevent regression, we established five mandatory benchmark invariants:
VERIFIED REPRODUCIBILITY