← AI Assurance
Case study · High-risk AI oversight

The model change nobody re-assessed

Illustrative composite — a fictitious infrastructure operator, simulated data. Built to show the mechanism, not to describe any real client or engagement. The same pattern runs live in our demo.
< 24 hrs
from unassessed retrain to blocking flag raised — vs ~90 days waiting for quarterly audit
1 diff
model registry vs assessment log — the entire detection mechanism
0 new tools
for the operator's ML team — the agent reads systems already in place

The challenge

A fictitious infrastructure operator runs an AI video-analytics system for restricted-area intrusion detection across 13 sites — a high-risk system under its assurance framework. Policy is unambiguous: a material change to a high-risk system triggers reassessment before continued operation. The obligation mirrors real regimes — the NSW AI Assessment Framework, ISO 42001 clause 8 change management, and the EU AI Act's "substantial modification" rules all say the same thing.

The problem: the obligation lived in a PDF. The ML team's retraining pipeline lived in a model registry. Nothing connected them. When the intrusion-detection model was retrained mid-year — new data, materially changed behaviour — the pipeline did exactly what it was built to do, and the policy did exactly what documents do: nothing.

Every governance failure in this story is boring. A planned retrain. A filed policy. Two systems that never talked. That's what makes it universal.

What happened

Jun 03Model v4.2 deployed. Retrained on new footage; detection thresholds shift. Registry records the version bump. No reassessment scheduled — nobody's job to notice.
Jun 03–04The silent window. A high-risk system operates without a valid assessment. Under NSW AIAF-style rules, this is a reportable compliance gap. Historically it would surface at the next quarterly audit — if the sample happened to catch it.
Jun 04Agent daily check runs. The assurance agent diffs the model registry against the assessment log: version v4.2 has no matching assessment. System classified high-risk → blocking flag raised, assurance score drops, executives see it on the dashboard the same morning.
Jun 04Corrective action drafted automatically. CA raised with owner, deadline, and the evidence pack attached: registry entry, prior assessment, the policy clause breached. Reassessment is scheduled before the audit cycle even knows the change happened.

The mechanism

No new platform, no workflow migration. The agent reads two systems the operator already had and tests one rule continuously:

Model registry⇄ diff ⇄ Assessment log→ Gap = flag→ High-risk = blocking→ CA + evidence pack

Every finding is traceable: this artefact, this framework clause, this gap. When a board member asks why the score dropped, the answer is a chain of evidence — not professional opinion.

The result

The gap that would have waited a quarter for an audit — or longer, if sampling missed it — was found, flagged, owned and scheduled for remediation inside a day. The reassessment happened while the change was fresh. The audit trail wrote itself.

That is the difference between having a policy and having assurance: a policy describes the obligation; assurance is the mechanism that checks it — continuously.

Why this pattern is spreading

The regulatory logic is global. The EU AI Act requires fresh conformity assessment on "substantial modification" of high-risk systems — retraining is the canonical trigger. The NSW AI Assessment Framework has applied to every NSW agency since January 2026. And continuous-compliance programs report 3-year ROI above 285% against periodic-audit baselines.

Watch this exact failure play out live

Our Command Centre demo replays the pattern — drift flag day 18, unassessed model change day 26, score falling off a cliff — on simulated data. Then see the agent run a clause-level audit in real time.

Open the Command Centre Watch the live audit Start with a two-week sprint