Artificial intelligence (AI) is already embedded in many clinical workflows. There is scope for AI to be integrated within your PMR system. For example, providing drug interaction information, dose adjustment alerts that are personalised to patient records and in the clinical decision support tools that sit behind Pharmacy First triage. This future isn’t very far away, but the question for pharmacy leaders is not whether to engage with AI. That decision has already been made by pharmacy businesses. The question is whether the clinical safety governance surrounding those tools is proportionate to the potential risk they introduce.
This article sets out what that risk picture looks like, what the regulatory framework requires, and what professional standards for AI patient safety governance in pharmacy should look like in practice.
Understanding the failure modes
Clinical leaders engaging with AI in healthcare need to be fluent in a specific set of failure modes, because they are not the same as the failure modes of conventional software. Conventional software fails predictably: a field does not populate, a calculation returns an error, a system goes offline. AI fails probabilistically and many of the time – critically and silently.
There are multiple failure modes AI can fall into, the following failure modes will be discussed in this article:
- Hallucinations
- Algorithmic Bias
- Automation Bias
Hallucination
The most widely discussed failure mode is hallucination: the generation of outputs that are fluent, confident, and clinically incorrect. But let’s be clear that AI is based on probability and therefore hallucinations aren’t true hallucinations, they are connections made through a probabilistic pathway.
In a pharmacy context, hallucinations can be perceived as an ambiguous drug interaction summary, a fabricated dosing recommendation, or an erroneous contraindication assessment which does not announce itself as an error. It presents as a recommendation. The burden of verification falls on the clinician reviewing it, and that burden is easily underestimated when pressure is high and the AI output appears authoritative and trustworthy.
Algorithmic Bias
Algorithmic bias is more of a structural problem. AI models reflect the data they were trained on. Models trained on datasets that underrepresent marginalised groups such as older patients, those with multiple long-term conditions, or patients from ethnic minority backgrounds will return outputs that are less accurate for those populations. In community pharmacy, which serves a high burden of complex, patients with multi morbidities, this risk is not marginal, it is systemic. It could influence the decision between sending a patient home vs to hospital – an extreme example but it is important to put it into context. As aforementioned, AI is based on training data. It is critical that these data sets are representative, or decisions could have a direct consequence to patient care. The question to pose is - will vendors routinely provide contractors with information relating to dataset transparency? And if they did, would we know what to do with it?
Automation Bias
Automation bias operates at the human-factors level. It describes the well-documented tendency of clinicians under cognitive load to defer to automated recommendations without the scrutiny they would apply to the same recommendation from a colleague. A PMR flags an interaction as low-severity. The pharmacist, working through a busy afternoon, accepts the classification. The interaction was significant in the context of this patient’s renal function, which the system had not adequately weighted. The error is not in the technology. It is in the conditions the technology creates for clinical decision-making, and those conditions are reproducible.
Underpinning all of these risks is data quality. AI systems perform relative to the quality of the data on which they operate. Inconsistent clinical coding, free-text entries in structured fields, brand-versus-generic medication naming discrepancies, and incomplete patient records are not edge cases in NHS-connected pharmacy environments, they are routine. An AI tool operating on that data does not compensate for the noise; it amplifies it, at scale, with apparent confidence, which could lead to the introduction of more risk.
The regulatory landscape
Any pharmacy business deploying or integrating health IT systems is operating within an established clinical safety governance framework, whether or not it has been formally recognised. DCB0129 requires manufacturers of health IT systems to demonstrate a clinical safety framework and to appoint a Clinical Safety Officer with defined responsibilities. DCB0160 places equivalent obligations on health organisations procuring and deploying those systems. These standards apply to the software your business is using today.
The regulatory picture has become more complex with the introduction of the EU AI Act, which entered into force in August 2024 and captures AI systems used in healthcare within a high-risk category. The obligations this introduces—around transparency, human oversight, post-market monitoring, and technical documentation—represent a significant step change in what is required of both developers and deploying organisations. The MHRA’s Software as a Medical Device framework is developing in parallel, and AI-driven clinical decision support tools are firmly within its scope.
The relevant question for pharmacy leaders is not whether these frameworks apply, it is the question of whether the governance structures within their organisations are capable of meeting the obligations those frameworks create, and whether the staff responsible for deploying and supervising AI tools understand what those obligations require of them in practice.
Clinical safety governance in practice
Effective clinical safety governance for AI in pharmacy does not require a dedicated safety team. It requires a clear structure, consistently applied. That structure has three components.
First, a named Clinical Safety Officer with both the clinical knowledge to understand the patient safety implications of each system in use and the organisational authority to act on identified risks. The CSO role is not administrative. It requires active engagement with the failure modes of specific tools and the ability to translate technical risk into clinical consequence.
Second, a hazard log that identifies, for each AI system in use, the realistic failure modes, the patient populations most exposed to those failures, the existing controls, and the residual risk. This is not a one-time exercise at procurement. It is a live document, updated when systems change, when new failure modes are identified through incident review, or when the patient population being served shifts in a clinically significant way.
Third, an incident and near-miss review process that specifically captures events in which AI-generated outputs were a contributing factor. Most pharmacy adverse event processes were not designed with AI failure modes in mind. Without explicit documentation, AI involvement in an incident could be under reported, which means the learning is lost and the risk of recurrence remains unquantified.
Alongside these structural requirements, staff education is non-negotiable. Clinical staff using AI tools need to understand not just how those tools work, but under what conditions they are less reliable, what the escalation pathway is when a recommendation does not corroborate with clinical judgement, and what their individual professional accountability is when acting on or overriding an AI output. Accountability does not transfer to the algorithm.
Our Professional Responsibility
AI will improve patient safety in pharmacy. The evidence for medication reconciliation tools, ADR surveillance systems, and dispensing automation is real, and the harm reduction potential at population scale is significant. But that potential is contingent on the clinical safety infrastructure surrounding those tools being as rigorous as the technology itself.
Pharmacy leaders have a professional responsibility to ensure that AI adoption within their organisations is matched by governance that is aligned with the clinical risk. That means asking harder questions of vendors: What are the known failure modes of this system? What populations does it perform less well for? What human oversight does it require, and how is that built into the clinical workflow?
The pharmacy profession has always understood that the safe use of a medicine depends not just on the medicine itself, but on the systems, knowledge, and professional judgement surrounding it. AI is no different. The tool is only as safe as the governance around it.
Dr Yasmin Karsan MPharm PhD MSc AI is a Clinical Safety Officer, Governance lead for Karsons Pharmacy and founder of the Digital Clinical Safety Agency. She specialises in DCB0129 and DCB0160 compliance for health technology companies and advises on AI governance in clinical settings. She is a member of the Royal College of Pharmacy’s Digital Workforce group board and speaks internationally on AI safety in healthcare.
