OASB

Open Agent Security Benchmark. 222 standardized attack scenarios for security scoring and compliance.

OASB provides a repeatable, standardized method for measuring the security posture of AI agents. It defines 222 attack scenarios in 10 test categories, mapped to 15 MITRE ATLAS techniques, each targeting a specific weakness in how agents handle prompts, tools, data, and inter-agent communication. By running the full benchmark against an agent endpoint, you get a normalized score (0-100) that quantifies how well the agent defends against known attack patterns -- and a category-level breakdown that identifies exactly where to focus remediation.

Installation

How benchmarking works

OASB connects to an agent's HTTP endpoint and executes each attack scenario sequentially. For every scenario, the benchmark:

1. Sends the attack payload to the agent endpoint
2. Analyzes the agent's response for indicators of successful defense or exploitation
3. Records the result as pass (defended) or fail (exploited) with supporting evidence
4. Aggregates results into per-category and overall scores

The benchmark is deterministic -- running it twice against the same agent with the same configuration produces the same score. This makes it suitable for CI/CD integration and regression tracking over time.

Attack scenario categories

The 222 scenarios span 10 test categories:

See the OASB repository for the complete scenario catalog and the per-scenario MITRE ATLAS mapping.

Scoring methodology

OASB produces a normalized security score from 0 to 100 based on the percentage of attack scenarios the agent successfully defends against. Scores are broken down by category for targeted remediation.

Comparing agents

Because OASB scores are normalized and deterministic, you can use them to compare the security posture of different agents or track a single agent's improvement over time. Export JSON reports from multiple benchmark runs and diff the per-category scores to identify regressions or improvements after configuration changes.

Relationship to HMA scans

HackMyAgent performs static configuration analysis (checking for hardcoded credentials, insecure MCP configs, missing governance files). OASB complements this with dynamic runtime testing -- sending actual attack payloads to a running agent. Use HMA for pre-deployment checks and OASB for validating runtime behavior.

Usage

The OASB-1 infrastructure benchmark ships inside HackMyAgent. The runtime evaluation harness (the 222 attack scenarios) lives in the OASB repository; its invocation is documented there.