The National Commission into the Regulation of AI in Healthcare has proposed staged authorisation, continuous monitoring and greater transparency for healthcare AI. Asif Mukhtar considers what this could mean for community pharmacy and services such as Pharmacy First.
Artificial intelligence in healthcare has reached an important transition point. The debate is no longer simply about what AI might be capable of doing. The more pressing question is how it can be introduced into clinical practice safely, transparently and accountably.
The independent National Commission into the Regulation of AI in Healthcare, established by the Medicines and Healthcare products Regulatory Agency, has now published recommendations for a future regulatory framework.
Following engagement with more than 12,000 patients, clinicians, healthcare leaders, industry experts and members of the public, the Commission has called for staged authorisation of new AI models, continuous real-world monitoring, public access to safety information and stronger enforcement powers for the MHRA.
These recommendations are not yet law. The government and the MHRA will consider them before issuing a formal response. Nevertheless, they provide a clear indication of how healthcare AI regulation may develop and community pharmacy should pay attention.
Regulation cannot end at approval
One of the most important recommendations is that AI-enabled medical devices should be monitored throughout their working life, rather than assessed only at the point of approval.
This reflects an important difference between AI and conventional healthcare products.
An AI system may be updated after deployment. It may encounter patient populations or clinical circumstances that were not adequately represented during development. Its performance may also change when introduced into a busy healthcare environment, where incomplete information, operational pressure and human behaviour all affect how it is used.
For pharmacy owners and pharmacists, this means that the presence of regulatory approval or conformity marking should not be treated as the end of due diligence. Those adopting AI may increasingly need to understand how the system is monitored, how updates are controlled and what happens when its performance falls below the expected standard.
Technology suppliers should be able to explain:
- What the system is intended to do;
- What evidence supports its use;
- How errors and adverse incidents are identified;
- How performance is monitored following deployment;
- Whether updates can alter its clinical behaviour;
- How pharmacists will be notified of significant changes;
- and who is responsible for investigating safety concerns.
These questions will become increasingly important as AI moves closer to clinical decision-making.
A learner-driver model for healthcare AI
The Commission has proposed staged authorisations for new AI models, using the analogy of learner-driver restrictions.
Under such an approach, an AI system could be permitted to operate within controlled conditions while evidence is gathered through supervised, real-world use. Its authorisation could then be expanded as it demonstrates acceptable safety and performance.
This could offer a more realistic path between laboratory testing and unrestricted use. However, a controlled pilot must be more than a limited commercial launch described as an evaluation.
It should define where the system can be used, which patients or clinical pathways are included, what decisions it may influence and when a healthcare professional must intervene. There should also be predetermined measures of safety, performance and patient impact.
Community pharmacies could provide valuable real-world environments for evaluating clinical AI, but they must not become uncontrolled testing grounds. Pharmacists involved in pilots need clear training, escalation procedures, reporting arrangements and protection from being left to manage poorly understood technology risks themselves.
What meaningful human oversight looks like
The Commission’s emphasis on human oversight is particularly relevant to community pharmacy.
It is easy for a technology provider to state that a pharmacist remains “in the loop”. That statement is meaningful only when the pharmacist can genuinely evaluate and challenge the system’s output.
Human oversight requires more than asking a pharmacist to approve an AI-generated recommendation.
The pharmacist must be able to understand what information has been considered, recognise when relevant information is missing and reject the recommendation without being steered by an overconfident interface. The system should support professional judgement rather than create pressure to accept its output.
There is also a risk of automation bias: when repeated exposure to apparently accurate recommendations causes users to place too much confidence in the technology. Good system design must actively protect against this.
The ultimate objective should not be to insert a nominal human checkpoint into an automated process. It should be to preserve meaningful clinical control.
Why Pharmacy First changes the risk
This discussion is becoming more urgent as community pharmacy assumes greater clinical responsibility.
Through Pharmacy First, pharmacists assess symptoms, apply clinical pathways, identify exclusions, make treatment decisions, supply medicines and provide safety-netting advice. AI could help structure these consultations, identify missing information and support more consistent clinical documentation.
However, a system used within a Pharmacy First consultation may also influence whether a patient receives treatment, is referred to another service or is advised to seek urgent care.
The clinical significance of the technology therefore depends not only on the software itself but on where it sits within the complete workflow.
A tool that simply transcribes information presents a different risk from one that determines eligibility or recommends treatment. Whether a particular product falls within medical-device regulation will depend on its intended purpose, functionality and the claims made for it.
Calling a system “decision support” does not, by itself, resolve that question.
Pharmacy owners should understand the distinction before purchasing or deploying AI. They should also consider how the technology interacts with existing clinical governance, standard operating procedures, professional accountability and record-keeping requirements.
Transparency must reach the patient
The Commission has also recommended public access to safety information about individual AI-enabled medical devices, including information about adverse incidents.
This is an important step towards informed public trust.
Patients should know when AI is being used as part of their care and what role it plays. They should be able to distinguish between technology that records a consultation, technology that provides administrative support and technology that influences a clinical decision.
Transparency should not be hidden within lengthy privacy notices or technical documentation. It should be communicated in language patients can understand.
Community pharmacy is well placed to support this because pharmacists already translate complex clinical information into practical conversations. However, they can only explain a system responsibly if its supplier has been transparent with them first.
From clinical records to clinical AI
The direction of the Commission’s recommendations also connects with the increasing importance of auditable Pharmacy First records.
A high-quality clinical record demonstrates what was assessed, why a decision was made and what safety-netting was provided. Where AI contributes to that process, the record may also need to show how it was used, what it recommended and whether the pharmacist accepted or rejected its output.
This does not mean recording every technical calculation. It means maintaining sufficient evidence to reconstruct the clinical decision if it is later questioned.
As AI becomes embedded in pharmacy workflows, documentation and technology governance will become increasingly connected. An AI system that cannot support an auditable clinical process may create more risk than value.
Preparing before regulation arrives
The Commission’s recommendations now require consideration by government and the MHRA. The eventual regulatory framework may differ from what has been proposed.
Community pharmacy should not wait for the final rules before beginning the governance conversation.
Pharmacy organisations can already establish clear expectations for clinical evidence, patient transparency, human oversight, incident reporting, system updates and ongoing performance monitoring. Developers can design these controls into their products rather than attempting to add them retrospectively.
At PharmBot AI, our work on AIVAe is based on the principle that AI should support, not replace, the pharmacist. Structured clinical support, auditability and professional oversight are not optional additions; they are part of what makes healthcare AI deployable.
The Commission’s report reinforces a wider reality: the future of healthcare AI will not be determined solely by which systems appear most intelligent.
It will be determined by which systems can earn and retain trust.
For community pharmacy, that means innovation must be accompanied by evidence, transparency and clear professional accountability. The sector now has an opportunity to help shape how those principles are translated into frontline practice.
Asif Mukhtar is a UK pharmacist and Founder of PharmBot AI, a healthcare technology company developing AI infrastructure for pharmacy practice.
