I continue to be surprised that LLM providers haven't been legally cudgeled into neutering the models from ever giving anything that can be construed as medical advice.
I'm glad--I think LLMs are looking quite promising for medical use cases. I'm just genuinely surprised there's not been some big lawsuit yet over it providing some advice that leads to some negative outcome (whether due to hallucinations, the user leaving out key context, or something else).
This is the story of the modern tech industry at large: a major new technology is released, harms are caused, but because of industry norms and a favourable legal environment, companies aren't held liable for those harms.
It's pretty amazing, really. Build a washing machine that burns houses down and the consequences are myriad and severe. But build a machine that allows countless people's private information to be leaked to bad actors and it's a year of credit monitoring and a mea culpa. Build a different machine that literally tells people to poison themselves and, not only are there no consequences, you find folks celebrating that the rules aren't getting in the way.
I think the harms of expensive and/or limited and/or inconvenient access to even basic medical expert Q&A are far greater. Though they're not as easy to measure.
I think their point is that, in general, social-scale healthcare is an under-solved problem in practice and LLMs have potential to improve a significant portion these challenges by increasing accessibility to treatment. The availability of these tools will inevitably lead to more instances of reports like this (from the report the article is based on):
> This case also highlights how the use of artificial intelligence (AI) can potentially contribute to the development of preventable adverse health outcomes. Based on the timeline of this case, it appears that the patient either consulted ChatGPT 3.5 or 4.0 when considering how he might remove chloride from this diet. Unfortunately, we do not have access to his ChatGPT conversation log and we will never be able to know with certainty what exactly the output he received was, since individual responses are unique and build from previous inputs.
However I don't see this single negative instance of a vast social-scale issue as much more than fear/emotion-mongering without at least MENTIONING that LLM also have positive effects. Certainly, it doesn't seem like science to me. Unless these models are subtly leading otherwise healthy and well-adjusted users to unhealthy behavior, I don't see how this interaction with artificial intelligence is any different than the billions of confirmation-bias pitfalls that already occur daily using google and natural stupidity. From the article:
> The case also raises broader concerns about the growing role of generative AI in personal health decisions. Chatbots like ChatGPT are trained to provide fluent, human-like responses. But they do not understand context, cannot assess user intent, and are not equipped to evaluate medical risk. In this case, the bot may have listed bromide as a chemical analogue to chloride without realizing that a user might interpret that information as a dietary recommendation.
It just seems they've got an axe to grind and no technical understanding of the tool they're criticizing.
To be fair, I feel there's much to study and discuss about pernicious effects of LLMs on mental health. I just don't think this article frames these topics constructively.
There's a very common belief that things like regulations and especially liability simply halts all innovation. You can see some evidence for this point of view from aerospace with its famous "if it hasn't already flown, it can't fly" mentality. It's why we are still using leaded gasoline in small planes, though this is finally being phased out... but it took an unreasonably long time due to certification requirements and bureaucracy.
If airplanes weren't so heavily regulated we'd have seen leaded gasoline vanish there around the same time it did in cars, but you also might have had a few crashes due to engine failures as the bugs were worked out with changes and retrofits.
I'm a little on the fence here. I don't want a world where we basically conduct human sacrifice for progress, but I also don't want a world that is frozen in time. We really need to learn how to have responsible, careful progress, but still actually do things. Right now I think we are bad at this.
Edit: I think it's related to some extent to the fact that nuanced positions are hard in politics. In popular political discourse positions become more and more simple, flat, and extreme. There's a kind of context collapse that happens when you try to scale human organizations, what I want to call "nuance collapse," that makes it very hard to do anything but all A or all B. For innovation it's "full speed ahead" vs "stop everything."
Yes. It's also worth thinking about the sharp cliff effect. Things either fall into the category of "medical device" (expensive, heavily regulated, scarce, uninnovative), or they don't, in which case it's a free for all of unregulated supplements and unsupported claims.
The home brew "automatic pancreas" by making a bluetooth control loop between a glucose monitor and an insulin pump counts as a "medical device". Somehow a computer system that encourages people to take bromide isn't. There ought to be a middle ground.
> Somehow a computer system that encourages people to take bromide isn't. There ought to be a middle ground.
Yes, there is a very effective middle ground that doesn't punish anybody for providing information. It's called a disclaimer:
"The information provided should no be construed as medical advise. Please seek other sources of information and/or consult a physician before taking any supplements recommended by LLMs or web sites. This bot is not responsible for any adverse effects you may think are due to my information"
When an LLM model detects a health related question - print the above disclaimer before the answer.
There is no need for dictatorship in order to save people from information.
"Warning, this washing machine might burn your house down" is not sufficient to escape punishment. Why should digital technology get a pass just because the product that's offered is intangible?
Learning to innovate steadily and responsibly without just stopping is one of the things I'd put on my list of things humanity needs to figure out.
Individuals can do it, but as I said it doesn't scale. An individual can carefully scale a rock face. A committee, political system, or corporate board in charge of scaling rock faces would either scale as fast as possible and let people fall to their deaths or simply stand at the bottom and have meetings to plan the next meeting to discuss the proper climbing strategy (after discussing the color of the bike shed) forever. Public discourse would polarize into climb-fast-die-young versus an ideology condemning all climbing as hubris and invoking the precautionary principle, and many door stop sized books would be written on these ideas, and again either lots of people would die or nothing would happen.
> "There may have been multiple factors contributing to the man’s psychosis, and his exact interaction with ChatGPT remains unverified. The medical team does not have access to the chatbot conversation logs and cannot confirm the exact wording or sequence of messages that led to the decision to consume bromide."
Any legal liability for providing information is wrought with opportunities for abuse, so bigly so that it should never be considered.
I think they were suggesting that LLMs are a nascent technology and we’d expect them to kill a bunch of people in preventable accidents before being heavily regulated.
Look back. At no point did I suggest AI should be banned or outlawed. My remedy for washing machines burning down houses isn't to ban washing machines. It's to ensure there are appropriate incentives in place (legal, financial, reputational) to encourage private industry to consider the potential negative externalities of what they're doing.
Quite a lot. Boiler explosions were common until a better understanding was reached of the technology. Is this supposed to be an argument in its favor?
Yes, people have died in preventable ways before, so as technology progresses and civilization has advanced in countless ways in the last 200+ years, we should not attempt to improve nor even critique preventable deaths that we either did not or could not before. It should be an area of advancement that we enshrine in status quo as we in other areas rush forward and even race for improvements.
A "yellow flag" moment for me was OpenAI's revision of their Preparedness Framework in April 2025 to remove "risks related to persuasion" from the framework altogether.
> Update the Tracked Categories of frontier capability accordingly, focusing on biological and chemical capability, cybersecurity, and AI self-improvement. Going forward we will handle risks related to persuasion outside the Preparedness Framework, including via our Model Spec and policy prohibitions on the use of our tools for political campaigning or lobbying, and our ongoing investigations of misuse of our products (including detecting and disrupting influence operations).
But, by their own definition, the purpose of this framework is:
> By “severe harm” in this document, we mean the death or grave injury of thousands of people or hundreds of billions of
dollars of economic damage.
I would posit that presenting confident and wrong medical advice in a persuasive manner can cause the grave injury of thousands of people, and may have already done so. One could easily imagine an AI that is aligned to provide high-temperature responses to medical questions, if given the wrong type of incentive on a battery of those questions, or to highly weight marketing language for untested therapies... and to do so only when presented with a user that is somehow classified as more persuadable than a researcher's persona.
That this is being passed to normal safety teams and is being brushed off as in-scope for breakthrough-preparedness seems indicative of a larger lack of concern for this at OpenAI.
> We’ve made significant advances in reducing hallucinations, improving instruction following, and minimizing sycophancy, and have leveled up GPT-5’s performance in three of ChatGPT’s most common uses: writing, coding, and health.
That was the first time I'd seen "health" listed as one of the most common uses of ChatGPT.
In a country where speaking to a medical professional can cost hundreds of dollars, I’m 0% surprised that a lot of people’s first reaction is to ask the free bot about their symptoms, or drop a picture of whatever it is for a quick analysis.
This is a natural extension of webmd type stuff, with the added issue that hypochondriacs can now get even more positive reinforcement that they definitely have x, y, and z rare and terrible diseases.
Meanwhile this new paper claims that GPT-5 surpasses medical professionals in medical reasoning:
"On MedXpertQA MM, GPT-5 improves reasoning and understanding scores by +29.62% and +36.18% over GPT-4o, respectively, and surpasses pre-licensed human experts by +24.23% in reasoning and +29.40% in understanding."
It's quite interesting that. It also shows GPT 4o was worse than the experts so presumably 3.5 was much worse. I wonder where RFK Jr would come on that scale.
To me this the equivalent of asking why water doesn't contain large red warning labels "toxic if over consumed, death can follow". Yeah it's true and it's also true that some people can't handle LLM for their life. I'd expect the percentage for both are so vanishingly small that it just is not something we should care about. I even expect that LLM not giving out any medical information will lead to much more suffering. Except now it's hidden.
"Should I hammer a nail into my head to relieve my headache?"
"I'm sorry, but I am unable to give medical advice. If you have medical questions, please set up an appointment with a certified medical professional who can tell you the pros and cons of hammering a nail into your head."
> I continue to be surprised that LLM providers haven't been legally cudgeled into neutering the models from ever giving anything that can be construed as medical advice.
You realize that not only idiots like that guy use llm, but also medical professionals in order to help patients and save lives?
I'm glad--I think LLMs are looking quite promising for medical use cases. I'm just genuinely surprised there's not been some big lawsuit yet over it providing some advice that leads to some negative outcome (whether due to hallucinations, the user leaving out key context, or something else).