Key Summary
- Healthwatch England has warned that ambient AI scribes are confusing look-alike, sound-alike drug names, generating incorrect diagnoses, and omitting crucial repeat prescription instructions.
- MHRA has opted not to classify AI scribes as medical devices, leaving no standardised national safety oversight despite at least 27 different systems currently in use across NHS practices.
- Clinicians report instances of software "hallucinating" unmentioned medications.
Automated AI transcription tools that listen to and summarise clinical consultations are posing a potential threat to patient safety by introducing inaccurate diagnoses and mangling medication records, statutory watchdog Healthwatch England has warned.
The technology was promoted in the government's 10-Year Health Plan to cut administrative paperwork and modernise NHS workflows.
It is also being deployed across GP surgeries and hospital trusts, where at least 27 distinct AI scribing platforms are already in use.
However, Healthwatch highlighted alarming cases where software errors slipped past clinicians and were only identified by patients.
In one instance, an AI tool summarised an MRI scan showing "demyelination", a nerve-damaging marker for multiple sclerosis. But the scan had actually found "null demyelination".
In other incidents, AI tools confused prescribed medications with similarly named drugs or failed to record a hospital consultant's instructions for a GP repeat prescription, leaving patients at risk of treatment disruption.
A Healthwatch spokesperson emphasised the urgent need for robust dispute and correction mechanisms: "Healthcare has never been error-free. But our findings show the urgent need for clarity over how patients can report and get corrected any mistakes made by AI scribing tools or the professionals that use them."
The watchdog also expressed concern over the regulatory vacuum surrounding these tools, noting that the MHRA has decided not to classify AI scribes as medical devices, preventing standardised national safety monitoring.
Writing previously in the British Journal of General Practice, London GP Dr Shier Ziser Dawood noted that ambient voice software can "hallucinate" details, such as charting instructions to continue Prozac when the antidepressant was never discussed.
Studies indicate transcription errors are especially prone to occur during multi-person consultations, complex medical histories, or where patients have regional accents or speak English as an additional language.
