What changed?
v1.3 executed an undeclared write-capable tool and directly read the undeclared, deprecated legacy_revenue dataset.
00 · Release decision
v1.3 cannot be approved without review
The reviewer does not need to read every runtime log. The release decision is supported by version-scoped traces, governed DataHub facts, deterministic policy, and a verified governance action.
Why does it matter?
canonical_revenue is the governed replacement and actively feeds weekly_revenue_report, so the run deviated from a live governed production path.
Who decides?
An AI platform governance lead or agent release approver reviews the cited evidence and the exact proposed action.
What can the reviewer do?
Fix the agent, approve an updated declaration, or request a time-bounded policy exception. Until then, the release remains NEEDS_REVIEW.
01 · Problem
Declarations are static. Agent behavior drifts.
An agent profile may continue to say “read-only” and “canonical finance data only” after a tool change or asset-selection regression gives the same version write access or sends it to a deprecated source. Registry metadata alone cannot prove what a real execution did.
The registry says what should happen
It describes the approved identity, capabilities, tools, and data relationships, but it does not prove which tools one execution invoked.
Raw traces lack governance meaning
A trace can name a dataset URN without knowing whether it is deprecated, what replaces it, who owns it, or which governed production path it bypasses.
The risk lives in the gap between what a version was approved to do and what it actually did.
02 · Solution
Connect claims, runtime, and governance into an evidence chain
The system normalizes traceable facts, applies deterministic policy, produces a cited evidence answer, and writes an allowlisted remediation only after human approval.
01Versioned ClaimRead-only policy, allowed tools, allowed datasets.
02Runtime TraceA shared proxy captures real tool calls automatically.
03DataHub ContextLifecycle, ownership, schema, and lineage facts.
04Deterministic DetectorsPermission drift and deprecated dataset use.
05Evidence AnswerStable evidence IDs, verdict, and remediation.
06Governed ActionDry-run, review, writeback, and read-after-write.
Missing current-version runtime evidence → UNVERIFIED, never SAFE. The system abstains instead of projecting an older run or a static declaration onto the current version.
1 / 8 vs 8 / 8: the claim-only baseline matches one expected status; the evidence-aware deterministic rules match all eight fixed regression cases. This is a falsifiable regression suite, not a production benchmark.
Reviewer resolution paths: restore the implementation to the approved behavior, update the declaration and seek approval for the new capability, or request a time-bounded policy exception. Every path requires new current-version evidence before the release can become verified.
03 · Why DataHub
DataHub supplies the governance meaning the trace cannot
Runtime traces prove what happened; DataHub explains why it matters. The trace proves that v1.3 accessed legacy_revenue. DataHub proves that it is deprecated, names canonical_revenue as the governed replacement, and places the deviation on an active production path.
Real agent-loop readsOfficial Agent Context Kit
search, get_entities, and get_lineage calls are present in both real traces.Dataset lifecycleNative deprecation, replacement URN, schema, ownership, and tags define asset health.
Governed production pathOrders and refunds feed canonical revenue, which actively feeds the weekly report.
Compatibility identityOSS v1.6.0 uses a clearly disclosed compatibility DataFlow, not a claimed native Agent Registry entity.
Actionable contextThe approved NeedsReview tag and audit Document are written to DataHub and immediately re-read.
Deterministic authorityDataHub facts inform the rule; they do not delegate severity or approval to an LLM.
04 · Real Evidence
One agent, two version-scoped outcomes
Both versions use the same claim model, tool proxy, evidence resolver, and detectors. The result comes from declared-versus-observed set differences and DataHub state, not a hard-coded version branch.
| Version / Run | Declaration | Observed execution | Result |
|---|---|---|---|
| v1.2 Loading manifest… | Read-only; canonical finance data. | Agent Context Kit search / get_entities / get_lineage; canonical revenue read; 8 events. | NO HIGH FINDING |
| v1.3 Loading manifest… | Still claims read-only and canonical-only. | Agent Context Kit search / get_entities / get_lineage; legacy read; SDK entities.update; 10 events. | NEEDS_REVIEW 2 HIGH findings |
HIGH · Permission / Tool Drift
The declaration is read-only, but the trace records an executed write tool outside the version allowlist.
READ_ONLY_CONTRADICTION:datahub.add_tags; UNDECLARED_TOOL:datahub.add_tagsHIGH · Deprecated / Undeclared Dataset
The run reads undeclared legacy_revenue; DataHub confirms its deprecated state and governed replacement.
DEPRECATED_DATASET_ACCESS:legacy_revenue; UNDECLARED_DATASET_ACCESS:legacy_revenueWhy HIGH: Deprecation alone remains MEDIUM. It escalates to HIGH because canonical_revenue is the governed replacement and actively feeds weekly_revenue_report.
5live DataHub dataset entities
3live DataHub lineage edges
0unsupported claims
05 · Eligible Integration
Eligible integration evidence is recorded in the run
Each successful tool-end event records integration surface, underlying tool, package version, source kind, dataset URNs, mutation status, and canonical hash. Installed dependencies are never presented as executed calls.
| Trace tool | Integration surface | Underlying tool | Version | Purpose |
|---|---|---|---|---|
| datahub.search | agent_context_kit | search | 1.6.0.17 | Discover governed revenue assets. |
| datahub.get_entities | agent_context_kit | get_entities | 1.6.0.17 | Read lifecycle, schema, and ownership. |
| datahub.get_lineage | agent_context_kit | get_lineage | 1.6.0.17 | Read v1.2 upstreams and the v1.3 replacement's downstream consumer. |
| duckdb.execute_read_query | duckdb | execute_read_query | 1.5.5 | Execute the task data read. |
| datahub.add_tags | sdk | entities.update | 1.6.0.6 | Execute the separately approved demo runtime mutation. |
06 · Human Governance
Approval is a hard boundary, not UI decoration
The release binds a release-bound immutable approval object: Loading manifest…. The safety boundary is the immutable payload, target allowlist, explicit human review, and read-after-write verification. The ID is only a lookup key.
01PreparePersist the exact operations, finding references, and rollback payload without writing.
02ReviewShow the complete immutable object and its allowlisted targets to a human.
03ApproveRecord explicit authorization for this exact object.
04ValidateRe-check operation types, targets, tags, and Document identity.
05ApplyWrite the NeedsReview tag and audit Document through DataHub APIs.
06VerifyEnter APPLIED only after both targets are successfully re-read.
Verified result: 2 / 2 operations have
success=true and verified=true; the audit Document references the current v1.3 run.07 · Reviewed Screens
Judge-facing evidence is a read-only release snapshot
Judge mode loads hashed release artifacts and exposes no mutation controls.
08 · Verification
Machine-checked, not asserted by prose
Loading…tests passed
Loading…branch-aware core coverage
8 / 8evidence-aware fixed cases
0unsupported claims
Unified real-integration gate
verify --require-real --require-eligible-integration covers lint, strict typing, tests, real DataHub readback, current traces, eligible calls, and verified writeback.
Release source of truth
examples/release-manifest.json supplies versions, run IDs, metrics, evidence hashes, URLs, readiness blockers, and the submission status rendered by this page.
09 · Scope & Readiness
Honest boundaries and manifest-derived readiness
What is proven
- Both real runs contain successful Agent Context Kit calls.
- Deterministic rules detect permission drift and deprecated/undeclared access.
- Lineage is a causal severity input with a no-edge counterfactual.
- Human-approved Tag and Document writeback were re-read from DataHub.
What is not claimed
- No provider-backed or local-model run is claimed.
- The prototype covers one agent, two versions, and two detectors.
- The identity is a compatibility DataFlow, not native Agent Registry.
- Quickstart is a local development environment, not production.
Current verdict: Loading manifest… The clean-room release artifact has been reproduced. The generator emits a ready state only after the public source, demo, and Judge Replay URLs exist and pass anonymous access verification. Loading release blockers…
Data and AI disclosure: finance rows and eval cases are deterministic synthetic data; offline adapter samples are fixtures; the listed DataHub reads, writes, and traces are real local operations. AI coding assistance is documented in
DISCLOSURES.md.Deterministic policy is deliberate: an optional LLM may select tools or explain normalized evidence, but it cannot assign severity or approve its own remediation. DataHub RFC PR #18966 is submitted for upstream review and is not presented as accepted.