AI governance is an organisational control, not an engineering feature

When an AI system causes harm, the first document anyone asks for is usually the AI policy. It almost always exists. It is often well written. And it is frequently the least useful document in the bundle, because a policy only shows what an organisation intended. What a court, regulator or tribunal needs to know is what the organisation actually did, and that evidence is spread across far more people than the engineering team.

That is the central point about AI governance that most implementations miss. It is tempting to treat it as an engineering problem: add evaluation tests, log the prompts, put a human-in-the-loop button on the screen. Those controls matter, and I will come to them. But on their own they govern the model, not the decision to use it, the data fed into it, the people relying on it or the contract under which it was bought. Effective governance needs the whole organisation to buy in, and it needs several stakeholders who each own a different part of the risk.

Who has to be involved

Taking ISO/IEC 42001 (the AI management system standard) and the EU AI Act together, the obligations fall naturally across these roles:

Stakeholder	What they own	Evidence I would expect to find
Board and senior management	Risk appetite, accountability, resourcing	Minutes approving AI use cases and risk appetite; a named accountable executive; budget for oversight
Business owner of the use case	Purpose, intended use, acceptable outcomes	A documented use case and success criteria; sign-off before go-live
Legal and compliance	Regulatory classification, contracts, liability	Risk classification under the AI Act; supplier terms; records of legal review
Data protection officer	Lawful basis, DPIA, automated decision-making rights	A DPIA covering the AI processing; handling of Article 22 UK GDPR rights
Information security	Threat model, access, data leakage	Threat models covering prompt injection and data exfiltration; access reviews; DLP rules for AI tools
Procurement and vendor management	Third-party models and services	Due diligence on providers; model and data-use terms; exit plans
HR and training	AI literacy and acceptable use	Training records (AI literacy has applied under Article 4 of the AI Act since February 2025); acceptable-use policy acknowledgements
Engineering and data science	Building, testing, monitoring	Model and prompt registries, evaluation results, logs, monitoring and incident records
Front-line operators	Human oversight in practice	Evidence that reviewers had the time, information and authority to override

When I review an AI-related dispute, the gaps are rarely in the engineering column. They are usually in the rows above and below it: a use case nobody formally approved, a DPIA written after launch, a vendor whose terms allowed it to train on customer data, or a “human in the loop” who was reviewing three hundred decisions an hour with no realistic ability to disagree.

What the engineering controls look like

Engineering still has to produce the evidence that the rest of the organisation relies on. The controls that make governance demonstrable are fairly well understood:

• A registry of models and prompts. Every model in use, its version or snapshot identifier, the provider, the system prompt and configuration, and who approved each change. Prompts are code and should be versioned like code.
• Evaluation suites as release gates. A fixed, versioned set of test cases covering accuracy, harmful outputs, bias indicators relevant to the use case and known failure modes, run automatically on every model, prompt or retrieval change, with thresholds that block a release.
• Event logging. For high-risk systems, Article 12 of the AI Act requires automatic logging sufficient to trace the system’s operation. In practice: inputs, outputs, model version, retrieved context, user, timestamp and any human override, retained for a defined period and protected from alteration.
• Human oversight hooks. Article 14 requires that people overseeing high-risk systems can understand the output, recognise automation bias and intervene or stop the system. That is a design requirement for the interface and the workflow, not just a policy statement.
• Monitoring and drift detection. Production metrics compared against the evaluation baseline, with alerts when behaviour moves. Model providers update hosted models, and an unchanged prompt can produce different results months later.
• Incident handling. A route for users and staff to report harmful outputs, a triage process, and a record of what was done.

The timetable for the AI Act’s high-risk obligations has moved, but the substance has not, and ISO/IEC 42001 certification is already being asked for in procurement. Organisations designing to these controls now will find the eventual requirements much easier to meet.

Why engineering-only governance fails

I see three recurring patterns where governance has been delegated to engineering alone.

The model is governed, the use is not. The team has a well-tested model, but nobody decided whether it should be used for the decision it is now making. A summarisation tool approved for internal notes quietly becomes the first pass on customer complaints, then on eligibility decisions. Each step was small, and nobody outside engineering was asked.

Shadow AI bypasses the controls entirely. Staff paste customer data into consumer chat tools because the approved tool is slower or blocked. Without acceptable-use training, DLP rules for AI services and procurement of a sanctioned alternative, the governed system covers a fraction of actual AI use. This is an estate control and HR problem as much as a technical one.

Oversight exists on the screen but not in practice. A review step is built, but the operators are measured on throughput, have no access to the reasoning or source data, and are never told how often the model is wrong. The logs then show near-total agreement with the model, which is evidence of automation bias rather than of oversight.

What this means in a dispute

When governance is challenged, I look for contemporaneous evidence from each stakeholder, not just a policy and a test report. The questions are practical. Who approved this use case, and on what information? Was the DPIA done before or after launch? What did the evaluation suite cover, and did it run on the version that was live at the time? Could the operator realistically have overridden the output? Was the system the organisation governed the same as the system that caused the harm?

An organisation that can answer those from its own records, across legal, data protection, security, HR and engineering, has governance. One that can only produce the policy has a document.