Why It’s Dangerous to Use AI for Medical Advice

Researchers warn that chatbots can deliver inconsistent diagnoses and miss urgent cases, raising concerns about patient safety as more people turn to AI tools for health guidance

Using artificial intelligence chatbots to seek medical advice can be “dangerous”, according to a new study that found the technology poses risks to patients because of its “tendency to provide inaccurate and inconsistent information”.

The research was led by scientists from the Oxford Internet Institute and the Nuffield Department of Primary Care Health Sciences at the University of Oxford and published in the journal Nature Medicine.

Dr Rebecca Payne, a co-author of the study, said it showed that “despite all the hype, AI just isn’t ready to take on the role of the physician”.

“Patients need to be aware that asking a large language model about their symptoms can be dangerous, giving wrong diagnoses and failing to recognise when urgent help is needed,” added Payne, who is also a general practitioner.

Researchers asked nearly 1,300 participants to identify possible health conditions and recommend a course of action across different scenarios. Some participants used large language model software to obtain a potential diagnosis and next steps, while others relied on traditional approaches such as consulting a GP.

After evaluating the responses, researchers found the AI frequently delivered a “mix of good and bad information” that users struggled to distinguish.

While chatbots now “excel at standardised tests of medical knowledge”, the study said their use as a clinical tool would “pose risks to real users seeking help with their own medical symptoms”.

“These findings highlight the difficulty of building AI systems that can genuinely support people in sensitive, high-stakes areas like health,” Payne said.

Lead author Andrew Bean of the Oxford Internet Institute said the findings showed that “interacting with humans poses a challenge” even for top-performing large language models.

“We hope this work will contribute to the development of safer and more useful AI systems,” he added.

Get the latest news and insights that are shaping the world. Subscribe to Impact Newswire to stay informed and be part of the global conversation.

Got a story to share? Pitch it to us at info@impactnews-wire.com and reach the right audience worldwide