Hidden harms: When chatbot health advice goes wrong

Clinical takeaway: Ask patients whether they have used AI chatbots for health advice when evaluating unexplained symptoms, treatment decisions, or medication changes, because chatbot-generated guidance may not be documented or available for clinical review.

Millions of patients now have around-the-clock access to health information from AI chatbots, but little is known about the downstream consequences of inaccurate advice. Unlike traditional medical errors, harmful chatbot interactions often occur on private platforms, leaving clinicians and regulators with limited visibility into what patients were told.

A perspective article in Nature Health argues that one of the greatest risks of AI-generated health information is not simply that it can be wrong, but that resulting harms may be effectively invisible. The authors describe a case in which a 60-year-old man seeking to reduce dietary chloride reportedly replaced table salt with sodium bromide after consulting a chatbot, later requiring hospitalization for bromide toxicity marked by hallucinations and paranoia.

According to the authors, chatbot-generated health information can confidently deliver inaccurate, oversimplified, or outdated recommendations. They contend that these errors are difficult to detect because conversations remain on company-controlled platforms, users may be unable to retrieve prior exchanges, and there are few formal mechanisms for reporting or investigating harmful advice. As AI-generated health content increasingly appears not only in dedicated chatbots but also in search-engine summaries and social media feeds, exposure may extend beyond people actively seeking medical guidance.

The paper calls for greater transparency and oversight, including user access to health-related chatbot conversations, systems for flagging and investigating potentially harmful outputs, clearer labeling of AI-generated health content, and independent monitoring of AI health advice. The authors also suggest policymakers consider extending malpractice-style accountability frameworks to AI companies that provide health guidance.

“We’re getting medical advice from these chatbots, but nobody, literally no one, is looking at this,” said co-author Kaicheng Yang. “If there’s an error or an issue, there’s just no way for people to know.”

Source: Yang K, et al. (2026 Oct 9) Nat Health. The invisible risks of AI-generated health information