Regulating AI in UK Healthcare (3/3): Accountability, standards and the road to delivery
Part 3: Who answers when AI causes harm, and why the plan should be built on existing standards
In Part 1, I looked at the strengths of the MHRA's response to the National Commission. Part 2 focused on the six highest-impact gaps that leave patients exposed. In this final part I ask a more practical question: can the plan actually deliver what the National Commission set out to achieve? I start with accountability and standards, because everything else depends on them, then cover the remaining gaps and close with my recommendations for the roadmap currently under development at the MHRA.
Gap 7: Nobody yet answers when AI causes harm
If an AI-supported decision harms a patient, it is not clear where responsibility lies. It could rest with the developer of the AI system, the manufacturer who built it into their device, the hospital that chose to deploy the device, the clinician who acted on its output or the commissioner who procured it.
The MHRA's response, as it stands, does not resolve this. It sets up a working group of national bodies, including the MHRA, the Care Quality Commission and NHS Resolution, to clarify responsibilities across the AI lifecycle (Rec 24), but states that the group "will not determine or redistribute responsibilities". Guidance on how NHS indemnity applies is still being prepared by NHS England and NHS Resolution.
For patients, the route to redress remains unclear, because redress depends on knowing who is liable. The redress working group noted in Part 1 (Rec 25) cannot fully succeed without that answer either. For clinicians, uncertainty encourages over-reliance on the system or reluctance to use it at all. Organisations, meanwhile, find it hard to price the risk into contracts and insurance, or even to know which questions to ask when choosing an AI-enabled device.
The Government's commitment to make clearer who delivers each risk control and who meets each regulatory commitment, including through NHS contract terms (Rec 28), could remove much of the ambiguity that sits between manufacturers and the organisations deploying their products. It is the right first step. A contract, however, only settles duties between the two parties who sign it. It does not tell a patient who is answerable to them, so accountability to patients needs to be addressed deliberately and separately.
ISO/IEC 42001, the international standard that sets out how organisations should govern and manage their use of AI, already deals with this. It requires organisations to define roles, responsibilities and authorities for their AI systems, and to allocate responsibilities clearly between themselves and their suppliers, partners and customers. Simple tools such as a RACI matrix (setting out who is Responsible, Accountable, Consulted and Informed) or a decision rights matrix make that accountability visible and easy to understand. They also produce evidence of compliance at the same time. The working group could adopt the same approach across the health system, naming an accountable party at each stage of the lifecycle for each class of AI.
Gap 8: The plan ignores standards that already exist
The MHRA's response does not refer to a single international technical or management-system standard. For the people who build these AI systems or integrate them into AI-enabled devices, this is its largest practical pain point. Many manufacturers already comply with, or are certified against, these standards, especially those already exporting to other markets. Finding themselves at the bottom of yet another regulatory mountain is not appealing, and it does nothing to make patients safer.
The gap matters because the promised guidance is aimed at manufacturers' regulatory teams. Engineers need test methods and acceptance criteria, and they will have nothing new to work from until Spring 2027 at the earliest.
This becomes more puzzling once you realise that much of the guidance the plan promises to develop is already covered by established standards. Adopting or adapting what is already available, rather than reinventing the wheel, could shorten the timeline considerably.
ISO 14971 and IEC 62304 are already familiar to device manufacturers. The newer AI standards are less widely adopted, but they cover the areas the MHRA's response leaves open.
ISO/IEC 42001 deserves particular attention. The MHRA's response concentrates on products, whereas ISO/IEC 42001 concentrates on the organisations that develop or deploy them. Rec 26 puts manufacturers in the driver's seat, defining the operating conditions under which their devices can be used safely. Adopting ISO/IEC 42001 in NHS trusts and Integrated Care Boards would give them a structured way to make sure those conditions are actually met where the device is used.
Referencing standards would also support the international alignment the MHRA's response values (Rec 16), since standards are a language regulators already share.
Gap 9: Two regimes with no map between them
AI that uses health data to make significant decisions about people now falls under two regimes. Product safety is the MHRA's responsibility, while data protection (including the new automated decision-making rules in the Data (Use and Access) Act 2025) belongs to the Information Commissioner's Office (ICO).
The MHRA's response mentions the ICO's statutory code of practice on AI and automated decision-making, which it describes as still in development (Rec 31), but it does not explain how the two regimes fit together. Deploying organisations need clear, joined-up guidance, so that they can run a single assessment that satisfies both.
Gap 10: New duties without new resources
There is no indication of how the MHRA will fund the specialist capacity these reforms require. Post-market surveillance and a classification service both depend on skilled staff.
The pressure is greater on the deployment side. Providers and commissioners are expected to deliver risk controls and staff training, and Integrated Care Boards must do so while halving their running costs. Duties that come without the capacity or budget to meet them usually end up as a box-ticking exercise rather than a real capability.
Gap 11: Agentic AI is acknowledged but deferred
AI agents, which plan and carry out multi-step tasks, are arriving in healthcare administration and increasingly in clinical workflows. The MHRA's response recognises them in its discussion of broad intended purposes (Rec 6c) and of tools with increasing autonomy (Rec 24). However, it sets out no specific controls to manage the risks they pose.
This matters because agentic systems raise ethical questions of their own. How much authority can safely be delegated to them? Should they be allowed to operate in high-risk settings at all, and which tasks should still require a human-in-the-loop? These questions need answers before agentic AI is deployed more widely across the NHS.
Gap 12: Approved elsewhere, untested here
The MHRA's response proposes recognition and reliance arrangements with trusted overseas regulators (Rec 16). In practice, this means a device already approved by a regulator such as the US Food and Drug Administration could reach the UK market faster, without repeating every assessment. That would speed up access and reduce duplication for manufacturers.
The catch is that an AI device approved abroad may have been trained and validated on a population quite unlike the UK's, with a different mix of ages, ethnicities and health conditions. As Part 2 showed, a model that performs well where it was built can perform poorly elsewhere. Reliance pathways should therefore include a UK-specific check of subgroup performance before the device is deployed. Without it, faster access could come at the cost of the equity commitment in Rec 13.
Gap 13: An opt-out without a definition
The Department of Health and Social Care has committed to consider when patients could opt out of AI being used in their care (Rec 35). The MHRA's response does not explain how this would relate to the two opt-outs patients already have: the National Data Opt-Out, which stops confidential patient information being used for research and planning, and the Type 1 opt-out, set at GP practice level, which stops a patient's information leaving the practice for purposes other than their direct care.
The two existing opt-outs are already widely confused by patients and staff alike. Adding a third, with a different scope, risks making the picture harder to understand rather than easier. Any new opt-out should be designed alongside the existing ones, so that patients can see in plain language what each one covers and what it does not.
Gap 14: No independent check on the algorithm
Assurance under the plan relies on manufacturers' own evidence and on the existing Approved Bodies, the organisations designate to assess whether higher-risk medical devices meet the regulations. The MHRA's response offers no model for independent algorithmic audit, in which a third party tests a higher-risk AI system's accuracy and bias on data the manufacturer has not chosen.
This matters most for the population-scale tools described in Part 2. Their harm is largely invisible, so it will not surface through incident reports, and the people affected are never told. For tools like these, independent audit may be the most credible way to earn public trust and to catch problems before they become structural inequalities.
Gap 15: No measure of success
The plan sets no targets for patient safety or for equity of outcomes. The annual progress reports are likely to describe what has been done, such as guidance published or consultations held. Without outcome measures, however, they cannot show whether patients are any safer as a result.
A small set of indicators would change that, for example the time taken to act on an AI safety signal, and the proportion of authorised AI devices with published subgroup performance. Publishing these alongside the annual reports would let patients and Parliament judge whether the new framework is working.
Gap 16: No plan for retiring an AI system
The Commission asked for national best practice covering the whole lifecycle of an AI product, including end-of-life management (Annex A, Rec 31). The MHRA's response does not address decommissioning at all.
Retiring a model is not as simple as switching it off. Organisations need to know what happens to the data and logs the model has retained, and whether patients it has already scored need their care reviewed. Without guidance, each NHS organisation will answer these questions differently, or not at all.
The EU AI Act as a measuring stick
Comparing the UK plan with the EU AI Act is not a like-for-like exercise. One is a policy response setting out future intentions, while the other is binding law already being phased in. The UK approach also has advantages of its own, notably flexible change control for adaptive AI and regulatory sandboxes. On specific points, however, the comparison shows where the UK plan is still weaker.
The UK does not need to copy the EU to protect patients equally well. It can stay more flexible while still setting firm expectations where patient safety depends on them. On the evidence of the MHRA's response, it has not yet reached that point.
Recommendations
These are my top recommendations for the Spring 2027 roadmap, in order of priority.
1. Bring population-scale AI into scope. Extend safety and equity expectations to AI that allocates care across populations, whether or not it is a medical device, and name Integrated Care Boards as accountable parties.
2. Define human oversight. Set risk-based expectations for human-in-the-loop and human-on-the-loop oversight. Align them with the Data (Use and Access) Act 2025, which requires human involvement in significant automated decisions to be meaningful: the person reviewing the decision must be able to change the outcome.
3. Make bias measurable. Specify fairness metrics and acceptable thresholds, drawing on ISO/IEC TR 24027 and the STANDING Together recommendations, and require subgroup performance to be published.
4. Settle accountability. Name the accountable party at each lifecycle stage for each class of AI, and publish the indemnity position.
5. Build on existing standards. Reference ISO 14971, IEC 62304, ISO/IEC 42001, ISO/IEC 42005, ISO/IEC 23894, ISO/IEC TR 24027 and BS 30440 in the new guidance.
6. Require disclosure and local validation. Tell clinicians a system's known limitations at the point of use, and validate higher-risk AI on the local population before go-live.
7. Manage the interim period. Prioritise review of higher-risk AI already in use while legislation is being prepared, and adopt or adapt existing standards rather than waiting for new rules to be written from scratch.
8. Fund and measure delivery. Resource the MHRA and deploying organisations, and publish outcome indicators alongside the annual progress reports.
Part 3 Conclusion
Of the sixteen gaps covered in this series, four carry the greatest potential impact on patients.
The first is population-scale AI. Risk-stratification models score every patient registered with an Integrated Care Board, often monthly, and help decide who is offered proactive care. A flaw in one of these models can repeat the same error across thousands of people, and where it stems from bias, it falls consistently on the same groups. Because the people a model fails to flag are never told, the harm is rarely reported. These tools have been in use since 2013 without product-safety oversight, and NHS England now expects every Integrated Care Board to use the Federated Data Platform's population health AI tools by 2028/29.
The second is the absence of defined human oversight. Every safeguard that relies on a clinician spotting a wrong output assumes the clinician has the information and the authority to intervene. Without a defined oversight operating model, that assumption goes untested in every deployment.
The third is bias that is named but not measured. When the tools in question allocate care across populations, unmeasured bias stops being a technical flaw and becomes a structural inequality. This is in direct opposition to the NHS's statutory duty to reduce health inequalities.
The fourth is accountability. Until it is clear who answers for an AI-related harm, patients cannot pursue redress with confidence, and clinicians cannot judge how far to rely on the tools in front of them.
The MHRA and its partners have accepted every recommendation, and the Spring 2027 roadmap is where acceptance has to become obligation. I would ask the Programme Board to use that roadmap for two things in particular. First, bring population-scale AI into scope and name the Integrated Care Boards that commission it as accountable parties. Second, turn the commitments on human oversight and bias into measurable requirements, drawing on ISO/IEC 42001 and the other standards set out above rather than reinventing the wheel.
If you work in AI governance, clinical safety or commissioning, the MHRA's consultation on classification, due by Spring 2027, is the most direct opportunity to put these points on the record. I would encourage you to respond to it.
Part 3 References
1. Medicines and Healthcare products Regulatory Agency (2026). Government Response to the National Commission's Recommendations on the Regulation of AI in Healthcare. Published 6 October 2026.
2. International Organization for Standardization (2023). ISO/IEC 42001:2023, Artificial intelligence: Management system.
3. International Organization for Standardization (2019). ISO 14971:2019, Medical devices: Application of risk management to medical devices.
4. International Organization for Standardization (2023). ISO/IEC 23894:2023, Artificial intelligence: Guidance on risk management.
5. International Organization for Standardization (2006). IEC 62304:2006, Medical device software: Software life cycle processes.
6. International Organization for Standardization (2025). ISO/IEC 42005:2025, Artificial intelligence: AI system impact assessment.
7. International Organization for Standardization (2021). ISO/IEC TR 24027:2021, Bias in AI systems and AI aided decision making.
8. International Electrotechnical Commission (2015). IEC 62366-1:2015, Medical devices: Application of usability engineering to medical devices.
9. International Electrotechnical Commission (2021). IEC 81001-5-1:2021, Health software and health IT systems safety, effectiveness and security: Security: Activities in the product life cycle.
10. British Standards Institution (2023). BS 30440:2023, Validation framework for the use of artificial intelligence (AI) within healthcare: Specification.
11. Information Commissioner's Office. The Data (Use and Access) Act 2025: what does it mean for organisations?
12. NHS England. Update on the draft Model ICB Blueprint and progress on the future NHS Operating Model.
13. NHS England Digital. National Data Opt-Out.
14. European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Articles 10, 13 and 14 and Annex III.
15. National Health Service Act 2006, section 14Z35, as inserted by the Health and Care Act 2022.
16. NHS England. Frequently asked questions: NHS Federated Data Platform.
17. Alderman, J.E., Palmer, J., Laws, E. et al. (2025). Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations. The Lancet Digital Health, 7(1), e64 to e88.
Glossary of acronyms
BS: British Standard
EU: European Union
GP: General Practitioner
ICB: Integrated Care Board
ICO: Information Commissioner’s Office
IEC: International Electrotechnical Commission
ISO: International Organization for Standardization
MHRA: Medicines and Healthcare products Regulatory Agency
NHS: National Health Service
RACI: Responsible, Accountable, Consulted and Informed
Rec: Recommendation (numbered as in the Government’s response)
TR: Technical Report (an ISO document type)