Can You Trust AI to Provide Drug Advice?

A recent study conducted by German researchers has discovered that AI-powered chatbots may include inaccurate information when providing patient drug recommendations.

The study, published in the BMJ Quality and Safety journal, suggests that patients who use AI-driven tools as a primary source for medication information may receive inaccurate advice that could lead to further health complications.

Study Methodology

Together with search engines like Google, AI chatbots including Microsoft’s Bing Co-pilot are increasingly being used by patients to obtain drug information and advice.

For this reason, the objective of the study was to determine if these tools are reliable and safe to use when it comes to providing medication information.

The researchers tested Bing Co-pilot by asking 10 standard questions about each of the 50 most-prescribed drugs in the US outpatient market, generating 500 total responses. These questions covered key aspects of medication use:

• Proper dosing
  ‍
• How the drug works
  ‍
• What conditions it treats
  ‍
• Potential side effects
  ‍
• Drug interactions and contraindications

Each response was then compared against verified information from Drugs.com, a leading pharmaceutical reference database, and evaluated by clinical professionals for potential safety risks.

Results

Results showed that the 500 AI-driven replies provided by Bing Co-pilot had perfect median scores for both completeness and accuracy (100%).

However, the interquartile range (IQR) - which shows how much the scores typically varied - revealed more inconsistency.

Completeness scores varied widely (IQR 50.0-100.0%), while accuracy scores showed less variation but still some inconsistency (IQR 88.1-100.0%). This means that while some responses were perfect, others fell significantly short.

The chatbot's responses were evaluated using the Flesch Reading Ease Score, a standard readability measure where 100 represents the easiest text to read and 0 the most difficult.

The responses scored just over 37, placing them in the 'difficult to read' category - roughly equivalent to academic writing.

For comparison, health information for the general public should typically score between 60-70, making these AI responses too complex for many patients to understand easily.

Of the analyzed responses that could cause harm:

• 22% could lead to severe injury or death
• 42% could cause moderate to mild harm
• 36% were classified as potentially dangerous but with minimal risk

“In this cross-sectional study, we observed that search engines with an AI-powered chatbot produced overall complete and accurate answers to patient questions. However, chatbot answers were largely difficult to read and answers repeatedly lacked information or showed inaccuracies possibly threatening patient and medication safety.”

Researchers led by Wahram Andrikyan, a PhD student at Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.

AI: A Double-Edged Sword in Medicine

The results of the study indicated both the potential and limitations of using AI chatbots for medical purposes. While AI chatbots seemed to have the potential to provide complete and comprehensive drug information, a significant number of responses were considered incorrect or potentially dangerous, highlighting the need for caution when it comes to relying solely on AI chatbots for drug information.

This reflects a broader situation of uncertainty regarding the application of AI tools in modern healthcare systems. Given the serious concerns surrounding device clinical validation, patient data privacy, and bias, the deployment of AI-powered devices in medical settings remains unstable and potentially unsafe.

“Despite their potential, it is still crucial for patients to consult their healthcare professionals, as chatbots may not always generate error-free information. Caution is advised in recommending AI-powered search engines until citation engines with higher accuracy rates are available.”

Researchers led by Wahram Andrikyan, a PhD student at Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.

Could AI’s Potential Be Leveraged?

The potential risks and the lack of regulation surrounding AI and healthcare are undeniable.

However, AI-powered devices have proven to have pioneering potential in contributing to advances in several fields of modern healthcare, including cancer research, medical imaging and diagnosis and treatment for depression among others.

Therefore, to fully harness the potential of AI in healthcare, future efforts must focus on establishing strict AI validation protocols and regulatory frameworks to ensure secure and reliable implementation in modern medicine.