OASB: Why AI Agents Need CIS-Style Security Benchmarks

The Problem: No Security Baseline for AI Agents

AI agents are shipping to production faster than security teams can assess them. These systems execute code, access databases, invoke APIs, read credentials, and communicate with other agents. The attack surface is fundamentally different from traditional software -- and the security tooling has not caught up.

OWASP published the Agentic AI Top 10 in early 2026, identifying risks like excessive agency, prompt injection, and insecure tool use. This is valuable work. But risk identification alone does not give teams what they need to actually secure their deployments.

What is missing: testable controls with audit procedures, remediation steps, and maturity levels. The equivalent of what CIS Benchmarks provide for operating systems, cloud infrastructure, and network devices -- but for AI agents.

What CIS Benchmarks Got Right

The Center for Internet Security (CIS) publishes over 100 benchmarks covering operating systems, cloud platforms, databases, and network devices. These benchmarks have become the de facto standard for infrastructure hardening because they share a common structure that makes them actionable:

This structure is what makes CIS Benchmarks automatable. Tools like CIS-CAT, OpenSCAP, and cloud-native policy engines can consume the benchmark, evaluate a system against it, and produce a compliance report -- all without human interpretation.

OASB applies this same model to AI agents. Every OASB control has the same four components: rationale, audit, remediation, and maturity level. This is not a whitepaper. It is a machine-readable specification designed to be consumed by automated scanners.

OASB Structure: 10 Categories, 46 Controls, 3 Levels

OASB-1 organizes 46 security controls into 10 categories. Each category addresses a distinct aspect of agent security:

Each control within these categories includes a unique ID, title, description, audit procedure, remediation guidance, and a mapping to relevant frameworks (OWASP Agentic Top 10, MITRE ATLAS, NIST AI RMF).

The three maturity levels allow teams to adopt OASB incrementally:

Why Machine-Readable Matters

Most security frameworks ship as PDFs. Teams read them, interpret them, build internal checklists, and hope their interpretation matches the auditor's. This process is slow, error-prone, and impossible to automate.

OASB is defined in YAML and JSON. Every control, every audit procedure, every remediation step is structured data. This design decision has three practical consequences:

Here is what an OASB control looks like as structured data:

# OASB-1 Control 5.3: No Hardcoded Credentials
id: oasb-1-5.3
title: No Hardcoded Credentials
category: configuration
level: L1
description: >
  Agent source code and configuration files must not
  contain hardcoded credentials, API keys, tokens,
  or other secrets.
audit:
  method: static-analysis
  check: scan source files for credential patterns
  tool: hackmyagent secure --check credentials
remediation:
  action: Move credentials to environment variables
  command: npx secretless-ai protect ./
mapping:
  owasp_agentic: A06-Inadequate-Sandboxing
  mitre_atlas: AML.T0040
  nist_ai_rmf: GOVERN-1.1

This is not a description to be interpreted. It is a specification to be executed. The audit section tells a scanner exactly what to check. The remediation section tells a developer exactly how to fix it. The mapping section tells a compliance team exactly which framework requirements it satisfies.

OASB v2: 222 Test Cases and Attack Simulation

The initial OASB-1 specification defined the controls. OASB v2 adds the test infrastructure to verify them at scale. The improvements fall into three areas:

Comprehensive test coverage. Each of the 46 controls now has multiple test cases -- positive tests (verify the control is met), negative tests (verify failures are detected), and edge cases (verify boundary conditions). The 222 test cases run against the OASB detection engine in under 3 seconds.

Attack simulation integration. OASB v2 connects benchmark compliance with real-world attack outcomes. HackMyAgent's attack mode sends 55 adversarial payloads (prompt injection, jailbreaking, data exfiltration, capability abuse, context manipulation) against an agent, then maps the results back to OASB controls. A passing L2 benchmark should correlate with higher resistance to these attack categories.

Compliance scoring that shows the path forward. Instead of a binary pass/fail, OASB v2 produces a score that shows both current posture and recoverable points. A score of 27/100 is not a failing grade -- it is a starting point with a clear path to 71 by addressing credential and authorization controls first.

$ npx hackmyagent oasb ./my-agent

OASB-1 Compliance Report
Level: L1 (Basic)
Score: 27/100

Category Breakdown:
  Identity:       3/5   [##---]
  Authorization:  1/5   [#----]  +12 recoverable
  Communication:  4/5   [####-]
  Data:           2/4   [##--]   +8 recoverable
  Monitoring:     0/5   [-----]  +15 recoverable
  Supply Chain:   3/4   [###-]
  Resilience:     2/5   [##---]  +9 recoverable
  Configuration:  1/5   [#----]  +12 recoverable
  Lifecycle:      3/4   [###-]
  Compliance:     2/4   [##--]

Path forward: 27 -> 71 by fixing Configuration
and Authorization controls first (+24 points)

Getting Started

Running an OASB benchmark scan takes one command. No configuration files, no account creation, no API keys.

# Run OASB-1 benchmark against your agent project
$ npx hackmyagent oasb ./my-agent

# Specify a maturity level
$ npx hackmyagent oasb ./my-agent --level L2

# Output as JSON for CI pipelines
$ npx hackmyagent oasb ./my-agent --format json

# Output as SARIF for GitHub Security tab
$ npx hackmyagent oasb ./my-agent --format sarif -o oasb.sarif

# Fail the build if score drops below threshold
$ npx hackmyagent oasb ./my-agent --fail-below 60

For CI/CD integration, add the benchmark as a step in your pipeline:

# .github/workflows/security.yml
name: OASB Compliance
on: [pull_request]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Run OASB-1 Benchmark
        run: npx hackmyagent oasb . --fail-below 60

      - name: Upload SARIF
        if: always()
        run: npx hackmyagent oasb . -f sarif -o oasb.sarif

      - uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: oasb.sarif

Related Reading