Regulating AI in UK Healthcare (2/3): The gaps that put patients at risk

Part 2: Six gaps in the Government's response that leave patients exposed to risks

In Part 1, I highlighted the strength of the recommended plan and gave the Government's response credit for several practical ideas. In part 2 and 3, I will point to the gaps and take the opportunity to propose how to address them in the roadmap which is currently under development. I'll first turn to the six gaps I consider most serious for patient safety, ordered by severity of the potential harm is and the size of population it could impact. The first is the largest, and it has been left open for more than a decade.

Gap 1: AI that decides who receives care across the whole populations

Every Integrated Care Board in England runs population health management, with risk-stratification models at its core. These models score each registered patient or resident on their likelihood of a future adverse health event, such as an emergency hospital admission or a need for high-cost care. The scores then decide who is offered proactive support to reduce or manage that risk.

Falls in older people are a good example. A model can estimate each older patient's risk of falling, so that those at highest risk are prioritised for a closer assessment of their frailty and living conditions, including a review of their medicines. Medicines matter here because of polypharmacy, usually defined as taking five or more medications at the same time. Taking several medicines together has long been linked to a higher risk of falls, and a study of more than 5,000 adults aged 60 and over in England found that people with polypharmacy fell 21% more often than those without it.

Acting early on these risks can prevent harm and reduce the cost of care. It also frees up hospital beds, a benefit that matters a great deal to hospital managers facing an ageing population. In 2024, occupancy of general and acute beds in England averaged 93.3%, well above the 85% level regarded as clinically safe, and people aged over 65 now account for 62% of hospital bed days. When beds are full, the effects spread through the whole hospital. In 2024, 1.7 million patients spent 12 hours or more in A&E, and surgeons warn that delayed discharges are holding back planned operations because there is no bed to admit patients into.

The falls-prediction model is a clear example of how AI used in population health management reaches well beyond a single clinical decision. Its outputs shape who receives proactive care and how scarce hospital capacity is used, so a single error can set off a domino effect that reaches patients the model never even scored. If the model misses older people at high risk, some of them will fall and be admitted to hospital. Each of those admissions takes a bed that a patient waiting in A&E, or for planned surgery, also needs. Yet this impact is neither regulated nor monitored as a safety risk.

The sums involved are large. For 2026/27, NHS England has allocated almost £126 billion to Integrated Care Boards for core services alone, and planning guidance expects population health tools, including risk stratification, to inform how those resources are allocated.

This is the gap that the MHRA's latest update brings into view. AI in medical devices is set to come under new, AI-specific rules, but AI models built or procured for population health management are not. These models are governed by data protection and equality law, which control how patient data may be used. No rules, however, test whether the model itself is accurate, unbiased or safe.

Part of the reason is that tools like the falls model sit on the boundary of what counts as a medical device. Under the established approach to qualifying software, a tool must be used for the benefit of individual patients to count as a device, while software that analyses population data does not qualify. A falls model does both. It is run across an entire population, but it produces a score that decides what happens to named patients. Whether it is a medical device is therefore unclear.

That ambiguity is exactly why it makes little sense to leave these tools without any proposed regulation, and the MHRA's response does not resolve it. MHRA limits the scope to medical devices, and risk-stratification software used for commissioning is generally not classed as one. MHRA's response also proposes exclusions for "administrative" and certain "decision support" software (Rec 1a), which may later confirm that population-level tools stay outside the proposed product regulation altogether.

Why scale changes the risk

A diagnostic device makes one judgement about one patient, usually with a clinician present who can notice when an output looks wrong.

A risk-stratification model scores an entire population in a single run (often hundreds of thousands of people, sometimes more than a million), and the run is usually repeated every month. A flaw in the model can therefore repeat the same error, in the same direction, for a large number of the people it scores. Where the flaw comes from bias, those errors fall consistently on the same groups of patients. A risk that would affect one patient in a single clinical encounter is multiplied across a whole group, and each monthly run repeats it.

Why the harm goes unseen

Most of the harm falls on people the model fails to flag, and they will very likely never realise that they were deprioritised. Because no clinician is involved when the score is produced, and nothing visibly goes wrong, no adverse incident is ever reported.

Much of the MHRA's currently proposed safety design depends on reporting, through Yellow Card and a stronger reporting culture. A system built on reports cannot detect harm that nobody is in a position to see.

Evidence from practice

A 2019 study in Science examined a population health algorithm widely used in the United States. It used past healthcare spending as a proxy for health need. Because less had historically been spent on Black patients with the same level of illness as white patients, the algorithm consistently under-identified them for extra care. The authors found that correcting the design would more than double the share of Black patients selected for additional support.

That algorithm was not a medical device either. It was a commissioning tool, applied to a whole population, with no product-safety oversight applied.

An old gap that is getting wider

Commissioners have used risk stratification since clinical commissioning groups were set up in April 2013. Within a year, NHS England had secured Section 251 approval for commissioning and GP data to be used for risk stratification, so that GPs, supported by their clinical commissioning groups, could target specific patient groups. Throughout the period since, these tools have been governed by data protection and confidentiality law, through lawful basis, Section 251 approval, opt-outs and impact assessments. Those controls establish whether the data may be used, but none of them tests whether the model is accurate or fair.

The gap is now widening. NHS planning guidance expects Integrated Care Boards to run risk stratification, and NHS England's Medium Term Planning Framework 2026/27 to 2028/29, published in October 2025, expects every board to use the Federated Data Platform's population health management tools by 2028/29. The platform has not been formally mandated, but use of these AI-based tools is fast becoming a national expectation while their assurance remains optional due to lack of enforced regulations.

The commissioners are left out of the MHRA's response

The MHRA's response assigns duties to manufacturers, providers, professionals and regulators. Integrated Care Boards, which buy these tools and act on their outputs, are not mentioned at all. Their statutory duty to reduce health inequalities (section 14Z35 of the NHS Act 2006) is precisely what a biased AI-based population model would undermine.

The EU comparison

The EU AI Act lists certain AI systems used by public authorities to evaluate people's eligibility for essential public services, including healthcare, as high-risk. A population model that determines who is offered NHS support could arguably fall within that category. The UK plan offers no equivalent classification or clarification.

What should change to address the highest-risk gap

As explained above, failing to regulate the AI systems used to manage healthcare at population level is the largest missed opportunity in the MHRA's latest response. Product-safety expectations should be extended to AI systems that allocate care at population level, whether or not they meet the definition of a medical device.

The roadmap could start with two safeguards that respond directly to the risks set out above. First, developers should publish how their models perform for different patient groups, so that any demographic biases are detected before deployment rather than years later, once evidence of harm has emerged. Second, accountability for these AI systems needs to be clarified and enforced. Each Integrated Care Board should name a board-level owner who is accountable for how these AI systems are used and how their risks are managed. That executive should then roll out effective AI governance throughout the downstream structure. These steps are the minimum requirements to close the gap left by the MHRA's silence on commissioners.

Gap 2: No defined role for the human

The MHRA's response does not say how people should oversee AI decisions. It sets no requirement for a human-in-the-loop, where a person reviews each output before it takes effect, or for a human-on-the-loop, where a person monitors the system as it runs and can step in when something goes wrong.

The closest it comes is a reference to governance of tools "with increasing degrees of autonomy" (Rec 24), and to training for staff who use tools with automated decision-making (Rec 33). Neither describes what effective oversight looks like in practice.

Having a person involved in the process does not guarantee oversight. Research on automation bias shows that clinicians tend to accept a system's recommendation, even when it is wrong. A reviewer who signs off an AI output almost automatically, as if the decision had already been made, offers very little protection.

UK data protection law has already moved on this point. The Data (Use and Access) Act 2025 sets a statutory test for meaningful human involvement in significant automated decisions that use health data: the reviewer must be able to change the outcome. The MHRA's response makes no connection to that test.

Similarly, Article 14 of the EU AI Act requires high-risk systems to be designed so that people can oversee them effectively. It names automation bias as a risk that overseers must be aware of, and requires the capability to override the system's output or stop it altogether. The UK plan should set an equivalent expectation, proportionate to the level of risk the system poses, and make clear which oversight model applies to which type of AI.

Gap 3: Bias is named but not measured

Part 1 gave the MHRA's response credit for naming the groups at risk of unequal performance (Rec 13). What it does not provide is a clarification on how a tester could tell whether a product meets that commitment.

•      There is no definition of fairness, and the common definitions can conflict with each other.

•      There is no metric, such as the gap in false-negative rates between groups.

•      There is no defined threshold for an acceptable gap.

•      There is no requirement to publish subgroup performance, so buyers cannot compare products.

Without these, two manufacturers could both claim compliance on the strength of entirely different tests. An equity commitment that cannot be audited is very hard to enforce.

The building blocks already exist.

First, ISO/IEC TR 24027, an international technical report on bias in AI systems, describes how bias arises and how it can be assessed.

Second, the STANDING Together recommendations add a layer specific to healthcare. Published in The Lancet Digital Health in December 2024, they are an international consensus, agreed by 194 experts from 25 countries, on how health datasets used to build AI should be documented and used. In short, they ask the dataset curators to be open about which population groups a dataset includes or excludes, and ask model developers to check whether those gaps lead to biased results downstream. The work was funded by the NHS AI Lab and The Health Foundation, so the UK has already invested in exactly the kind of guidance the MHRA's response does not mention.

The guidance due in Spring 2027 should draw on both.

Gap 4: Soft commitments during an admitted gap

Although the Government accepts that the current regulations are not fit for purpose, only two of its 44 responses commit to a deadline: draft change-control guidance by December 2026, and a classification consultation by Spring 2027. The remaining 42 promise "further details" by Spring 2027, and most of them are framed softly, as options to "explore" or "consider".

Those are reasonable words for a policy response, but they do not create obligations. Secondary legislation can only follow consultation, so the rules the Government itself describes as unfit are likely to stay in force well into 2027 and possibly beyond. The MHRA's response offers nothing for that interim period, such as a priority review of higher-risk AI already in use.

The EU AI Act already creates binding duties for high-risk AI, even though they apply in phases. The UK has chosen flexibility over certainty, which is defensible provided the interim period is actively managed.

It is understandable that a full overhaul is a large task that needs prioritising. It can, however, be delivered faster by building on what already exists. The MHRA does not need to reinvent the wheel. Instead, it could start by reviewing existing ISO standards and AI risk management frameworks (which Part 3 sets out), then decide which parts to adopt as they are and which to adapt for UK healthcare, rather than building everything from the ground up. Given the risk of harm to patients while regulation is delayed, this is the better path.

Gap 5: Clinicians are not told where the AI fails

The MHRA's response promises guidance on model cards and dynamic labelling (Rec 9) and commits to transparency for patients (Rec 35). In both cases, the commitment is only to explore how transparency might work, not to require it. The MHRA will consider how to help manufacturers explain how their devices work. The Department of Health and Social Care will consider how patients should be told when AI is used in their care. Neither places a duty on manufacturers to tell the clinician, at the point of use, what the system cannot do or where its outputs cannot be relied on. As the response stands, it requires no transparency on an AI system's known failure modes, or on the patient groups for whom it performs less accurately.

This information already exists in manufacturers' technical files. The safety question is whether it reaches the clinician deciding whether to act on an output, and the organisation deciding whether to buy the AI-enabled device in the first place.

Article 13 of the EU AI Act requires instructions for use to state a system's accuracy and its known limitations. A similar requirement in the new UK device regulations, obliging manufacturers to disclose accuracy and known limitations to the clinicians and organisations that use their products, would cost little (or nothing for devices already compliant with the EU AI Act). It would also give the human oversight discussed in Gap 2 something to work with.

Gap 6: No check that the model works locally

A model that performs well where it was developed can perform poorly somewhere else, because patient populations and clinical workflows differ between hospitals and regions.

This has happened at scale, and well-documented examples have already been published in peer-reviewed journals. In 2021, an independent evaluation at a US academic hospital found that a widely deployed sepsis prediction model performed far worse than its developer had reported, missing around two-thirds of the sepsis cases it was meant to detect.

The MHRA's response does not require deploying organisations to validate a model on their own population before go-live. Silent local evaluation (running a model in the background without acting on its outputs) appears only as a question about whether such testing counts as research (Rec 44). Site-specific tuning is mentioned as something change-control plans should allow (Rec 6b), but tuning a model and checking that it works are different activities.

BS 30440, the British Standard validation framework for AI in healthcare, already describes how validation should be approached. For higher-risk AI, local validation should be a condition of deployment.

Part 2 Conclusion

The six gaps share a cause. The MHRA's response was designed around medical devices that are used with one patient at a time and whose failures come to light through incident reports. Much of the AI now used in the NHS does not work that way.

The most serious risks sit where that assumption fails. Population health tools score entire populations, often every month, yet sit outside product regulation, and the Integrated Care Boards that commission them are not mentioned in the MHRA's response at all. Human review is assumed rather than specified. A clinician is expected to approve an output without knowing where the system is most likely to fail. Bias is acknowledged but never clearly measured or compared against a threshold, and a model can be deployed without its performance first being assessed on the population it is expected to serve.

None of this needs to wait for regulation to be written from scratch. Much of what is missing already exists, in international standards and in UK-funded work such as the STANDING Together recommendations. The MHRA could adopt or adapt these existing components to manage current risks sooner, reducing harm to patients and wasted resources across the UK healthcare system.

The gaps discussed here lead to a harder question. When a population model deprioritises the wrong patients, or a clinician acts on an output nobody warned them about, who is accountable? Part 3 shows that the MHRA's response leaves this key question unanswered. It also looks  at how existing standards could help close these gaps today.

Part 2 References

1.     Medicines and Healthcare products Regulatory Agency (2026). Government Response to the National Commission's Recommendations on the Regulation of AI in Healthcare. Published 6 October 2026.

2.     NHS England. Population health management.

3.     Polypharmacy and falls in the elderly: a literature review (2014). PubMed Central.

4.     Sharma, S., Fida, S. and Soriano, F. Impact of polypharmacy on falls risk in elderly (>65 years): a balancing act. British Geriatrics Society conference poster.

5.     Dhalwani, N.N. et al. (2017). Association between polypharmacy and falls in older adults: a longitudinal study from England. BMJ Open.

6.     Royal College of Emergency Medicine (2025). Ministerial Briefing: Road to Recovery, Urgent and Emergency Care. June 2025.

7.     Royal College of Physicians. A failing system: RCP urges government to fix social care crisis amid surge in delayed discharges.

8.     Royal College of Surgeons of England (2025). Surgeons: failure to tackle delayed discharges will make meeting the 18-week target "feel like wading through quicksand".

9.     NHS England (2025). ICB allocations 2026/27 to 2028/29 (core services). Updated 17 December 2025.

10.  In Compliance Magazine. Software as a Medical Device, Part II (summary of the MEDDEV 2.1/6 qualification decision steps for standalone software).

11.  Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447 to 453.

12.  Health and Social Care Act 2012 (established clinical commissioning groups from 1 April 2013).

13.  NHS England (2014). Risk Stratification Assurance Statement Letter to CCG clinical leads, February 2014 (Section 251 approval CAG 7-04(a)/2013).

14.  NHS England. Risk Stratification Assurance Statement. Updated December 2021.

15.  NHS England. Frequently asked questions: NHS Federated Data Platform (sets out the Medium Term Planning Framework 2026/27 to 2028/29 expectation).

16.  UK Parliament. Written question 102980: NHS England: Databases, answered by the Department of Health and Social Care, 12 January 2026.

17.  National Health Service Act 2006, section 14Z35, as inserted by the Health and Care Act 2022.

18.  European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Articles 13 and 14 and Annex III.

19.  Goddard, K., Roudsari, A. and Wyatt, J.C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121 to 127.

20.  Information Commissioner's Office. The Data (Use and Access) Act 2025: what does it mean for organisations?

21.  International Organization for Standardization (2021). ISO/IEC TR 24027:2021, Bias in AI systems and AI aided decision making.

22.  Alderman, J.E., Palmer, J., Laws, E. et al. (2025). Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations. The Lancet Digital Health, 7(1), e64 to e88.

23.  Medical Devices Regulations 2002 (SI 2002/618), as amended.

24.  Wong, A. et al. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8), 1065 to 1070.

25.  British Standards Institution (2023). BS 30440:2023, Validation framework for the use of artificial intelligence (AI) within healthcare: Specification.