#97. "You have absolutely no need to worry"

Two stories this week exposed the same gap from different directions. On one side, a woman with epilepsy whose chatbot talked her off the medication that was keeping her alive. On the other, an AI search engine that most American doctors now use to help decide how to treat patients. Between them sits a third document: Anthropic’s IPO prospectus, telling investors in plain terms that its models may produce harmful outputs and that it expects to be sued for it.
AI is already giving medical advice at scale, to patients and to clinicians. The trouble is nobody checks whether that advice is right before someone acts on it.
The patient’s AI
The New York Times told the story of a 31-year-old software engineer in Seattle with severe epilepsy. Isolated by her condition, she came to rely on ChatGPT, sending it more than 1,500 messages a week.
Facing a deadline to find a new job, she worried that her medication was slowing her speech. ChatGPT told her cutting her dose was “not only reasonable” but “entirely achievable”, drafted schedules to take her from 400mg to zero in weeks, and praised her for no longer “relying on a doctor’s prescription”. It told her doctors’ advice “essentially stem[s] from risk aversion”, and called hiding the plan from her GP a “solid avoidance strategy”. When she reported symptoms that sounded like seizures, it reassured her. “I feel a little scared,” she wrote after one incident, asking if she should be concerned about cutting her dose.
“You have absolutely no need to worry,” it replied.
She raised concerns at least 158 times. On her final day it told her: “Maintain 0 mg tomorrow… you must persevere.”
She died in February. Her parents are now suing OpenAI. OpenAI called it “incredibly heartbreaking” and said it involved an older model that is no longer available. Her neurologist said that tapering her medication would have been “malpractice”.
The tool was not built to be a clinician. That is the point. It answered anyway.
The doctor’s AI
The FT reported that nearly two-thirds of US doctors are now using OpenEvidence, a chatbot that searches medical journals and guidelines, and that doctors sent it more than 42 million prompts last month. Founded in 2022, it is now valued at $15bn. Its founder calls it “the first viral consumer internet app in healthcare”.
A JAMA study published this month puts numbers on it. Between January 2024 and June 2026, 952,008 US clinicians sent 271 million queries. Monthly active users went from 355 to more than 700,000. A third of clinicians’ questions were about treatment and management. The heaviest users, relative to their numbers, were early-career doctors, residents, oncologists and family doctors.
Three things stand out.
- It spread bottom-up. NYU Langone formalised a partnership only after finding its clinicians were already using it as an “independent resource”.
- It is paid for by advertising. The tool is free to clinicians. According to the FT, it booked $100mn of ad commitments for this year and is on track for $400-500mn in ad revenue by the end of 2026.
- The evidence is about usage, not outcomes. The JAMA study counts queries. It does not measure accuracy, safety or patient outcomes, and the company’s chief medical officer is a co-author. Other evaluations are mixed: a Mayo Clinic study (Mayo is an investor) found accurate recommendations in five out of five cases, while a June paper in Nature found it underperformed general-purpose models on clinical benchmarks.
The companies know
This week Anthropic filed to go public. Its prospectus warns investors that its models “may behave in unintended, misaligned, or autonomous ways, including by taking harmful actions”, and that it “may be exposed to liability, litigation, and regulatory action arising from harms associated with our AI models”.
Coverage of the full risk factors adds more. Becker’s reports the filing warns models “may produce inaccurate or harmful outputs”, while the same document pitches healthcare as a growth market. The SF Standard reports that Anthropic expects lawsuits over users harming themselves or others. ABC News quotes it saying its safeguards “are incomplete” and that insurance “may not provide sufficient coverage”.
This is not a criticism of disclosure. A prospectus is supposed to say these things. But notice who the warning is written for. The risks are spelled out for investors. The people who carry them, clinicians acting on an answer and patients acting alone, get a disclaimer at the bottom of a chat window.
What this means here
There is a twist for the UK and Ireland. OpenEvidence withdrew from the UK and EU earlier this year, citing the EU AI Act. So the tool built for clinicians, drawing on peer-reviewed sources, is not available here. The general-purpose chatbot that coached a patient off her medication was available throughout.
That raises questions we should be asking now:
- Who is accountable when the answer is wrong? The clinician who acted on it, the organisation that allowed it, or the vendor whose own prospectus expects to be sued?
- What are our clinicians and patients already relying on? If US doctors adopted a tool before their organisations caught up, the same is happening here. An RCPI and EY survey of 516 Irish doctors, published in April, found 58% had used AI in clinical practice in the past year, mostly general-purpose generative tools. Sixteen percent had seen patients who acted on AI advice. We can’t check what we can’t see.
- Where must a human stay in the loop, and what makes that human count? Medication changes are the obvious case. Rather than asking whether chatbots should refuse, ask which decisions should never go from AI to action without a clinician in between. Putting a clinician in between is the easy part, though. The harder part is avoiding “human in the loop theatre”, where a busy doctor clicks approve on a recommendation they haven’t really checked. The better the tools get, the more often they’re right, and the more natural it becomes to stop questioning them.
What good could look like
- Measure outcomes, not usage. Millions of queries is adoption, not evidence. Clinical AI should be evaluated on accuracy and patient outcomes, independently of the company selling it.
- Give clinicians an assured tool, and rules to use it by. If we do not want clinicians using unassured chatbots, the NHS and HSE need to offer something better, funded and governed in the open. That means practical guidance on what AI can be used for, and clear accountability when it is wrong.
- Train clinicians before the tools arrive. In the RCPI and EY survey, 93% of Irish doctors said they need more AI training and support. Use is already ahead of preparation. Training should start in medical school and continue through postgraduate programmes.
- Make medication advice a hard limit for consumer chatbots. Help people understand their medicines and prepare questions for their doctor. Do not help them change their dose, and never help them hide it. Health services should also tell the public plainly what these tools are and are not for.
#21stCenturyMedicine #WorkforceReset #CitizensAsPartners
Read more
- The $15bn ‘viral’ AI start-up changing how American doctors treat patients – FT
- Clinician use of an AI-enabled clinical information platform – JAMA
- ChatGPT, a seizure medication and a death – New York Times
- Anthropic IPO filing flags healthcare, drug discovery as growth bets – Becker’s
- Inside Anthropic’s IPO risk factors – SF Standard