ENGINEERING NOTE / AI AGENTS
A config file does not tell you what an AI agent is actually running.
Agent configuration is an input to runtime state, not proof of runtime state. The distinction matters once model routing, instructions, permissions and tools can all be resolved dynamically.
THE TRAP
Configuration describes intent. It does not necessarily describe the effective agent.
It is tempting to answer “what is this agent running?” by opening its configuration file. That works only if every relevant choice is already explicit there and no later layer changes it.
AI-agent runtimes often have more moving parts: a model can be selected by provider defaults, project instructions can be discovered from the workspace, permissions can be materialized when a session starts, and an MCP server can expose a different capability surface without the application repository changing.
A config file is still valuable evidence. The mistake is treating it as the strongest possible evidence for a runtime question it cannot answer on its own.
THREE PLANES
Declared, resolved and live should be separate claims.
I find it useful to split agent state into three evidence planes.Declared is what configuration and deployment inputs say should be used.Resolved is what the runtime produces after applying its own precedence and defaults.Live is what can be observed directly from the environment at collection time.
Those layers can agree, but they do not have to. More importantly, evidence from one layer should not silently be promoted into another. If a model name is missing from configuration, guessing the likely provider default does not make it resolved evidence.
This is the same reason I keep “missing” different from null in diagnostic data. Unknown and explicitly empty are different facts, and collapsing them removes information exactly where debugging needs it most.
RESOLUTION
When possible, ask the runtime that owns the resolution rules.
Reimplementing another tool's configuration precedence is attractive because it looks deterministic. It is also a good way to build a second resolver that eventually disagrees with the real one.
While building RuntimeTruth, I used Codex's own app-server path for canonical workspace configuration and then created a new ephemeral thread when I needed deeper effective state. The thread starts no model turn, but it causes Codex to materialize state for that workspace.
In real validation, the workspace configuration left model and provider fields unset. The ephemeral thread resolved a concrete effective model/provider together with approval, sandbox and instruction-source state. The useful fact existed only at the resolved layer.
LIVE EVIDENCE
Observation is stronger only for the thing actually observed.
Live evidence still needs a boundary. If Codex reports an instruction-source path and a verifier hashes the file currently present there, the hash proves the bytes observed at that path at that moment.
It does not automatically prove that those exact bytes were later placed into every model request. Likewise, seeing an MCP tool catalog tells you what capability surface was exposed through the inspected runtime; it does not tell you whether the server's contract is safe or backwards-compatible.
“Live” should therefore mean directly observed, not omniscient. Narrow evidence with honest provenance is more useful than a stronger-sounding statement the tool cannot actually establish.
DRIFT
The useful question is often not “did anything change?” but “did this invariant change?”
Once a runtime snapshot includes enough evidence, strict equality becomes noisy. A CLI version can change without invalidating the model or instruction policy you actually care about.
That is why I prefer explicit protected invariants over immediately inventing a large policy language. A deployment gate can protect the effective model, sandbox and instruction fingerprints while allowing unrelated evidence to move.
The failure modes matter too. A protected value that differs is drift. An invalid selector or an evidence comparison that cannot be made faithfully is an error. Neither should quietly become a green result.
DESIGN RULE
A green result should make one precise claim.
Runtime verification becomes dangerous when “matched the baseline” gets translated into “secure”, “compliant” or “safe”. Those are much broader claims.
The claim I want is smaller: the selected evidence matched the selected baseline under known comparison rules. That leaves room to reason separately about whether the baseline is trusted, whether the evidence coverage is sufficient and whether the resulting agent behavior is acceptable.
This evidence-first split is the core idea behind RuntimeTruth, but it is also a more general debugging habit: say what the source proves, keep resolution separate from intent, and stop the conclusion where the evidence stops.
RuntimeTruth
Evidence-backed runtime verification and drift detection for AI agents.