RuntimeTruth

The repository was not enough to answer what the agent was actually running.

RuntimeTruth came out of operating self-hosted AI workers and hitting a simple systems question: if the source tree looks unchanged, can I still prove which model, instructions, permissions and tools are effective right now?

For an ordinary service, process identity and Git state already tell part of that story. An AI agent adds another mutable layer: provider resolution, project instructions, sandbox policy and MCP capabilities can all change independently of the application code.

I started the project deliberately from evidence I could establish on a real runtime, rather than designing a policy platform first and hoping the data model would fit later.

Declared, resolved and live are different claims.

The central design decision was to stop treating “configuration” as one flat truth. RuntimeTruth records declared state, resolved effective state and live observations separately, with provenance attached to every evidence record.

That distinction matters because a field can be absent from the workspace config and still become concrete when the runtime starts real work. It also keeps the tool honest: an inference is not silently promoted into a live observation just because it looks plausible.

The schema stays intentionally small and deterministic. Missing stays different from null, snapshot capture time is not runtime drift, and provenance changes are rejected when schema v1 cannot compare them faithfully.

Use the runtime's own resolver instead of reimplementing its precedence rules.

The first agent-aware adapter targets Codex. Rather than parsing every config file and guessing which layer wins, RuntimeTruth talks to the Codex app-server and asks it for canonical workspace configuration.

When deeper effective state is required, the CLI can create a new ephemeral thread without starting a model turn. That probe materializes the effective model/provider, approval policy, sandbox state and instruction-source paths for the inspected workspace.

In real validation, the workspace-level config left model and provider unset while the ephemeral thread resolved them. That was the point where the project proved its thesis: the resolved layer contained useful runtime identity that a config dump alone did not.

Fingerprint the thing that matters without turning the verifier into a data collector.

Codex reports instruction-source paths. RuntimeTruth fingerprints the bytes it can read at those paths with SHA-256, but does not serialize the instruction plaintext.

The claim is deliberately narrow: the hash proves which bytes RuntimeTruth observed at a Codex-reported path at snapshot time. It does not claim cryptographic proof of the exact prompt bytes later consumed by a model.

The same restraint applies around MCP. RuntimeTruth keeps status, bounded tool-name inventory and deterministic full-catalog fingerprints while avoiding raw environment values, auth tokens, resource contents, descriptions and schemas.

A useful baseline gate needs to tolerate the right drift, not all drift.

Strict snapshot comparison was the first useful gate, but real runtimes contain state that can change without invalidating the thing you care about. I added exact protected invariants so CI can say, for example: let the Codex CLI version move, but keep the effective model, sandbox and instruction fingerprints fixed.

$ runtimetruth verify baseline.json --codex . --resolve-thread \
    --protect codex.thread.model \
    --protect codex.thread.sandbox \
    --protect codex.instructions

PASS: runtime matches baseline.

The selector path is intentionally small rather than a speculative policy DSL. Unknown fields fail as errors instead of creating a dangerous silent PASS, and the same semantic diff engine backs human output, strict verification, selective verification and the JSON reporter.

The first end-to-end test was a changed instruction file.

I captured a real Codex baseline, changed the same AGENTS.md file in place, and verified the workspace again. RuntimeTruth reported only the instruction fingerprint change and returned the dedicated DRIFT exit code.

The selective gate then demonstrated the opposite case: protecting only the effective model ignored that unrelated instruction change and returned PASS. A deliberately invalid selector returned ERROR rather than silently passing.

That small three-way validation — PASS, DRIFT and ERROR on a real runtime — was more useful to me than adding several unvalidated agent adapters for surface area.

The agent and the MCP server are different verification targets.

RuntimeTruth records the MCP capability surface visible to the inspected agent because that surface is part of the agent's effective runtime identity. It does not try to classify whether an MCP server's schema change is backwards-compatible or whether a tool description looks like a rug pull.

That distinction became clearer when I compared the project with MCPWard. MCPWard is a strong MCP contract-testing tool; RuntimeTruth asks a different question: did the effective agent runtime consuming those dependencies change?

Keeping that boundary explicit stopped the project from expanding into a worse duplicate of an adjacent tool and sharpened the product around agent runtime identity.

Small surface area, strong evidence boundaries.

The runtime package stays dependency-free outside the Python standard library. CI covers Python 3.12 and 3.13, builds both sdist and wheel, then installs the wheel into a clean environment and smoke-tests the CLI.

GitHub Actions are pinned by commit SHA. Current collectors use narrow allowlists near sensitive data. MCP probing is explicit because asking Codex for server status can contact configured servers or refresh authentication, even though RuntimeTruth itself does not call MCP tools.

The public v0.1.0 release intentionally has no hosted control plane, telemetry, account system, paid support plan or SLA. The local CLI has to be useful on its own.

A green result should make one precise claim.

RuntimeTruth PASS means that the selected evidence matched the selected baseline under the comparison semantics implemented by that version.

It does not certify that an agent is secure, compliant or behaviourally safe. The caller also owns the baseline: v0.1.0 does not sign it, establish approval authority or make it tamper-evident.

I prefer those explicit limits to a broad “agent security” claim the tool cannot actually prove.

The same evidence-first pattern shows up elsewhere.

Read the product case study, inspect the source or try the release.

RuntimeTruth is maintained as a Sidelobe open-source project. The Sidelobe case study focuses on the product and its verification model; this page focuses on the engineering decisions behind it.