The Compliance Evidence Layer: Why Scanning Isn’t Enough

By /

Compliance automation promised to save us. Instead, it created a new category of manual work. Here’s what’s actually missing—and how to fix it.

The compliance industry has a dirty secret: most of what we call “automation” isn’t.

Scans run on schedule. Reports generate. Dashboards update. It looks automated. But underneath, security teams are still drowning in manual work—interpreting findings, writing POA&Ms, gathering evidence, preparing for audits. The tools find issues. Humans do everything else.

This isn’t a tooling failure. It’s an architecture problem. And it can’t be solved by better scanners, smarter dashboards, or more comprehensive GRC platforms.

What’s missing is an evidence layer—infrastructure that produces cryptographically verifiable proof of compliance state, continuously, without human intervention.

The Gap Nobody Talks About

Consider what happens when a vulnerability scanner runs:

  1. Scanner executes checks against your infrastructure
  2. Scanner produces a report of findings
  3. Gap begins here
  4. Human interprets which findings matter
  5. Human maps findings to compliance controls
  6. Human writes POA&M entries
  7. Human tracks remediation
  8. Human verifies fixes
  9. Human gathers evidence for auditors
  10. Human explains evidence to auditors

Steps 4-10 consume hundreds of hours per quarter. They don’t scale with automation. They scale with headcount.

This is why organizations with mature security programs still hire compliance analysts. This is why audit prep remains a fire drill. This is why “continuous monitoring” usually means “we run scans continuously, then do everything else quarterly.”

The scanner did its job. The human work never went away.

Reports Are Not Evidence

Here’s the uncomfortable truth: a report is a claim, not proof.

When an auditor asks for evidence that access controls are properly configured, what do most organizations provide?

  • Screenshots from a dashboard
  • PDF exports from a scanner
  • Spreadsheets compiled by analysts

These artifacts share a common problem: they require trust. The auditor must trust that the screenshot wasn’t doctored, that the export reflects reality, that the spreadsheet wasn’t manipulated. There’s no independent verification. No cryptographic proof. No chain of custody.

This isn’t evidence. It’s testimony.

Real evidence has different properties:

  • Verifiable: Someone outside your organization can confirm it’s true
  • Traceable: You can prove when, where, and how it was collected
  • Time-bound: It reflects a specific moment, not a vague “current state”
  • Machine-readable: Downstream systems can consume it without human translation
  • Tamper-evident: Any modification is detectable

Most compliance artifacts fail every single criterion.

The Missing Layer

What’s needed isn’t another scanner or another GRC platform. It’s infrastructure that sits between your systems and everything that consumes compliance data—producing evidence that stands on its own.

This is the Compliance Evidence Layer: a distinct architectural component with three responsibilities.

1. Policy as Data

Security intent becomes explicit, portable, and auditable.

Instead of configuring scanners through GUI wizards or writing imperative scripts, you define what should be true declaratively. A policy states: “SSH must not permit root login.” It doesn’t specify how to check—that’s the execution layer’s job.

This separation matters because:

  • Policy authors don’t need to understand scanner internals
  • Policies can be version-controlled, reviewed, and shared
  • The same policy works across different execution environments
  • Auditors can read policies without reading code

2. Secure Execution

Validation happens within strict, auditable boundaries.

The execution layer operates under constraints that make it trustworthy:

  • Contract-based: Collectors and executors are registered and whitelisted
  • Deterministic: Same policy + same state = same result, always
  • Provenance-bound: Every data collection is traceable to its source
  • Read-only: The system observes state, never modifies it

This isn’t about convenience—it’s about trust. When execution is constrained and auditable, the results can be trusted without trusting the operator.

3. Verified Evidence

Every evaluation produces cryptographically signed attestations.

An attestation isn’t a report. It’s a structured proof containing:

  • What policy was evaluated
  • What criteria were checked
  • What the outcome was
  • When the evaluation occurred
  • Cryptographic signature for integrity

Attestations are designed for machine consumption. They flow into SIEMs, feed threat models, trigger automated remediation, and satisfy auditors—without human translation at each step.

What This Changes

When evidence generation is automated and cryptographically verifiable, several things become possible that weren’t before.

Audits Become Queries

Instead of spending weeks gathering screenshots and stale reports, you query the evidence layer. What was the SSH configuration on this host at this time? Here’s the signed attestation. 3PAOs can verify independently. You’re always assessment-ready.

Drift Becomes Visible

When a control fails, you know immediately—not at the next quarterly review. Compliance drift becomes a security event, detected and alerted like any other anomaly. SOAR playbooks can execute automatically. Passing attestations close incidents with cryptographic proof.

POA&Ms Write Themselves

Attestations contain the exact control, exact failure, and exact evidence. There’s no interpretation needed. AI agents can consume this deterministic data to generate POA&Ms that are specific, actionable, and traceable to source evidence.

Threat Models Get Real Data

Attack path analysis typically runs on assumptions: “if this control fails, this path opens.” With continuous attestations, you know which controls are actually failing. Threat models update in real-time based on actual posture, not theoretical posture.

The Technical Foundation: ESP

This architecture requires a policy language designed for evidence generation—not adapted from scanning tools.

Endpoint State Policy (ESP) is an open-source framework we built specifically for this purpose. ESP separates security intent from execution logic, producing structured evidence that maps directly to compliance frameworks.

Here’s an ESP policy that maps to FedRAMP AC-6 (Least Privilege):

META
    esp_id `win-sam-database-protected`
    version `1.0.0`
    control_framework `FedRAMP`
    control `SV-253400`
    control_mapping `FedRAMP:AC-6`
    title `SAM database must be protected from unauthorized access`
    description `The Security Account Manager (SAM) database contains 
                 sensitive credential information. It must be protected 
                 and not readable by standard users.`
    platform `windows`
    criticality `critical`
META_END

DEF
    OBJECT sam_database
        path `C:\Windows\System32\config\SAM`
    OBJECT_END

    STATE protected_system_file
        exists boolean = true
        is_system boolean = true
        owner_id string = `S-1-5-18`
    STATE_END

    CRI AND
        CTN file_metadata
            TEST at_least_one all
            STATE_REF protected_system_file
            OBJECT_REF sam_database
        CTN_END
    CRI_END
DEF_END

This policy is readable by anyone. You don’t have to be a developer to understand what it’s checking. Compare that to SCAP/XCCDF—XML formats that require specialized tooling just to read.

The policy expresses intent: the SAM database should exist, be marked as a system file, and be owned by the SYSTEM account. The execution engine handles collection and validation. The result is a signed attestation proving the control’s state.

From Intent to Evidence: The Full Lifecycle

Once an ESP policy exists, the execution lifecycle produces evidence at every stage.

Execution with Attribution

ESP doesn’t just say “this passed” or “this failed.” It records who ran the check, what system it ran against, and exactly when it happened. The agent has an identity and version. The endpoint has an identity and platform. The timestamps are explicit.

This matters because evidence without attribution isn’t evidence—it’s just output. Identity is what makes results trustworthy and reusable, especially outside the tool that generated them.

Signed Results

Once ESP has evaluated the policy, it produces a cryptographically signed result. The observed evidence and the evaluation outcome are hashed, and that hash is signed by the agent.

That signature binds together:

  • The policy that was evaluated
  • The method used to collect data
  • The system it was collected from
  • The exact moment in time it was observed

Here’s what a real ESP result looks like (from an actual Windows 11 endpoint):

json

{
  "envelope": {
    "result_id": "esp-result-188a7b18ddf98f94",
    "agent": {
      "id": "esp-agent",
      "version": "1.0.0"
    },
    "host": {
      "id": "host-726b5c5c7c8d5ef7",
      "hostname": "FLAVORTOWN",
      "os": "windows",
      "arch": "x86_64"
    },
    "started_at": "2026-01-28T03:25:50Z",
    "completed_at": "2026-01-28T03:25:50Z",
    "content_hash": "sha256:fa19672c7e0ffdbc...",
    "evidence_hash": "sha256:66735dc4257c7aff...",
    "signature": {
      "algorithm": "tpm-ecdsa-p256",
      "signature": "gyttlYkeNStbzYKNkXYx...",
      "signed_at": "2026-01-28T03:25:50Z",
      "covers": ["content_hash", "evidence_hash"]
    }
  },
  "policies": [{
    "identity": {
      "policy_id": "win-sam-database-protected",
      "control_mappings": [{"framework": "FedRAMP", "control_id": "AC-6"}]
    },
    "outcome": "fail",
    "findings": [{
      "finding_id": "f-c003e941",
      "severity": "high",
      "title": "file_metadata validation failed",
      "expected": {"is_system": "true", "owner_id": "S-1-5-18"},
      "actual": {"is_system": "false", "owner_id": ""}
    }],
    "evidence": {
      "data": {
        "file_metadata_sam_database": {
          "exists": true,
          "is_system": false,
          "readable": false,
          "writable": false
        }
      },
      "collection_metadata": [{
        "collector_id": "filesystem_collector",
        "method": {
          "method_type": "file_stat",
          "description": "Query file metadata via Windows API",
          "target": "C:\\Windows\\System32\\config\\SAM"
        }
      }],
      "collected_at": "2026-01-28T03:25:50Z"
    }
  }]
}

At this point, the output isn’t a report or a log. It’s an attested statement about system state. If the result changes later, it’s obvious. If someone asks how the conclusion was reached, the evidence and method are right there.

Why This Matters for FedRAMP 20x

FedRAMP is explicitly moving toward machine-readable, continuously generated evidence that can be validated and reused—not screenshots or narrative explanations. Signed results are what make that possible.

The split ESP enables is exactly what FedRAMP 20x expects:

  • A stable SSP that declares intent
  • Continuously generated evidence in OSCAL Assessment Results that proves reality

OSCAL Integration

With the signed result in hand, moving into OSCAL is straightforward. There’s no interpretation step—just transformation.

The ESP result already contains everything the OSCAL Assessment Results model expects:

  • What was assessed
  • Which control it maps to
  • What was observed
  • How it was collected
  • When it happened
  • Which system it applies to

Instead of scraping logs or summarizing output, you’re projecting structured data into a standardized format:

json

{
  "assessment-results": {
    "uuid": "ar-188a7b18ddf98f94",
    "metadata": {
      "title": "ESP Endpoint Assessment Results",
      "oscal-version": "1.1.2"
    },
    "results": [{
      "uuid": "result-win-sam-database-protected",
      "start": "2026-01-28T03:25:50Z",
      "end": "2026-01-28T03:25:50Z",
      
      "reviewed-controls": {
        "control-selections": [{
          "include-controls": [{"control-id": "ac-6"}]
        }]
      },
      
      "observations": [{
        "uuid": "obs-sam-file-metadata",
        "title": "SAM database file metadata",
        "methods": ["TEST"],
        "collected": "2026-01-28T03:25:50Z",
        "props": [
          {"name": "path", "value": "C:\\Windows\\System32\\config\\SAM"},
          {"name": "is-system", "value": "false"},
          {"name": "owner-id", "value": ""}
        ],
        "remarks": "Collected via Windows API file_stat"
      }],
      
      "findings": [{
        "uuid": "finding-sam-not-protected",
        "title": "SAM database not protected",
        "status": "failed",
        "target": {"type": "control", "target-id": "ac-6"},
        "related-observations": [
          {"observation-uuid": "obs-sam-file-metadata"}
        ]
      }]
    }]
  }
}

OSCAL Assessment Results aren’t meant to store reports. They’re meant to carry evidence. Because ESP produces evidence natively, the mapping is lossless.

How This Fits the SSP

ESP doesn’t replace the SSP—and it doesn’t generate one SSP per endpoint. That would defeat OSCAL’s design.

Instead, the SSP stays declarative. You describe how endpoints are supposed to be managed using reusable components. A component representing “Managed Windows Endpoints” declares how controls like AC-6 are implemented at a design and process level. That component references the ESP policy as the authoritative definition of expected state.

The SSP says: “This is how we intend to implement and enforce this control across endpoints.”

ESP’s role is to continuously validate that claim.

The OSCAL Assessment Results generated from ESP prove the SSP remains true over time. They show, in a machine-readable way, whether endpoints actually match what the SSP declares—and when they don’t, those findings roll directly into POA&Ms.

This separation is intentional:

  • The SSP doesn’t churn every time an endpoint changes
  • Evidence updates continuously
  • Authorization decisions are based on current, verifiable state—not stale documentation

At that point, OSCAL stops being “a document format” and becomes what it was designed to be: a transport layer for intent and proof, with ESP supplying the proof.

Key Design Principles

Fail-Fast Compiler

Invalid policies are rejected at compile time, before reaching endpoints. The compiler enforces strict validation:

  • Syntax and grammar validation
  • Type compatibility checking
  • Reference resolution and cycle detection
  • Security limits baked into the binary (SSDF compliant)

Errors are caught before execution, not during a scan.

Constrained Execution

The runtime engine operates within strict boundaries:

  • Whitelisted command execution only
  • Deterministic evaluation order
  • Repeatable evidence generation
  • No code injection from policy files

Framework-Agnostic Mapping

One policy can map to NIST, FedRAMP, CIS, and MITRE simultaneously. Define the technical control once, map it everywhere, and produce the evidence each framework demands.

Who Needs This

Not every organization needs cryptographically verifiable compliance evidence. But if you recognize yourself in any of these scenarios, the current model is costing you more than you realize:

You’re pursuing FedRAMP, FISMA, or CMMC and audit prep consumes your quarter. Your 3PAO keeps asking for more evidence. You’re never quite “assessment-ready.”

You’re scaling infrastructure and compliance work scales with headcount, not automation. Every new system means more manual verification, more evidence gathering, more analyst hours.

Your security team is burned out on compliance theater—work that doesn’t improve security, just proves you did security. Your best engineers are gathering screenshots instead of improving defenses.

You have compliance tools but no compliance confidence. Dashboards show green, but you can’t actually prove controls are working. Auditors ask questions your tools can’t answer.

The Path Forward

The compliance industry spent twenty years optimizing scanning. We got faster scans, more comprehensive coverage, better dashboards. What we didn’t get was evidence.

That’s not a criticism—scanning is necessary. But scanning solves the wrong problem. Finding issues is easy. Proving compliance state continuously, verifiably, without human intervention—that’s what was never built.

The Compliance Evidence Layer isn’t a better scanner. It’s the missing infrastructure that makes scanning useful.

If you’re tired of compliance theater—if you believe security teams should improve security, not gather screenshots—we should talk.

ScanSet is building the Compliance Evidence Layer. ESP, the policy engine at its core, is open source on GitHub. We’re working with design partners to shape the platform. If this resonates, let’s have a technical conversation.

Scroll to Top