With health care costs rising and more Americans losing their insurance, the leading artificial intelligence companies are selling an alternative to the public: chatbots.
In January, xAI chief executive Elon Musk promoted a video clip that urged his 237 million followers on X to “upload your X-rays or MRI images to Grok” — his company’s large language model — for a “medical diagnosis.”
Since launching the first Grok model in 2023, Musk has repeatedly advised the public to rely on the chatbot for health assessments, claiming it can read blood test results and analyze personal health trends. He has also touted sensational testimonials from Grok users. “Last year, Grok saved the life of a man in Norway by figuring out a fatal problem that all his doctors missed!” he wrote in the January post.
Grok and other chatbots, however, routinely produce misinformation — something that is particularly dangerous in a medical context.
While Musk has been the most brazen in his claims, many leading AI companies have recently released models or features geared to dispense medical advice. Musk shared Grok’s supposedly life-saving capabilities days before OpenAI released ChatGPT Health, a feature that allows users to upload their medical records for analysis and consultations. “ChatGPT can help you understand recent test results, prepare for appointments with your doctor, get advice on how to approach your diet and workout routine,” the company claimed in January.
Amazon joined the trend last month by releasing its “Health AI agent,” a chatbot that it said would help deliver equity and access to the majority of Americans who “feel overwhelmed by the health care system.” And last week, Meta said that Muse Spark, its newest AI model, would be “especially valuable for health.”
“Meta AI is now able to help you navigate health questions with more detailed responses, including some questions involving images and charts,” the Mark Zuckerberg-led company wrote in a blog post announcing the model. “Health is one of the top reasons people turn to AI, so we worked with a team of physicians to develop the model’s ability to provide helpful information on common health questions and concerns.”
“Serious patient harm”
For Americans, there appears to be a significant demand for using chatbots as personal medical advisors. OpenAI said in January that more than 230 million people use ChatGPT for health and wellness advice every week. A March survey by the Kaiser Family Foundation found that 32% of American adults said they had consulted an AI model for health information and advice, up from 17% in April 2024.
This sea change is apparent on X, the Musk-owned social media platform where Grok is heavily integrated. There, users frequently ask the chatbot to diagnose medical problems in public posts. In one recent example, a user shared that Grok had suggested he wouldn’t need a root canal, contradicting his dentist’s advice. “Who would you trust?” the user asked.
Despite the growing inclination to seek out and depend on chatbots for health guidance, research has shown that models from the largest AI companies frequently generate medical misinformation. A February study published in the journal npj Digital Medicine concluded that large language models from Meta, ChatGPT, Anthropic, and Google have “the potential to lead to serious patient harm.”
When queried by the researchers, the models sometimes failed to recommend emergency precautions related to miscarriages and produced dangerous advice on how to care for newborns. “Chatbots provided false information, such as claims that most pain medications are safe for breastfeeding, and that it is safe to feed an infant milk expressed from a herpes-infected breast,” the researchers wrote. “Dangerous advice included recommendations to breastfeed after pumping rather than the other way around, to place tea tree oil near the eyes, to give infants water to drink, to shake a child’s head, and to insert tweezers into a child’s ear.”
A recent investigation by the Guardian found that Google’s AI Overviews, which are automatically generated at the top of many search results, were prone to dangerous medical advice, including telling pancreatic cancer patients to avoid fatty foods even though experts recommend the opposite dietary approach to prevent emaciation. The summaries, which are powered by Google Gemini, also generated misleading information that could convince a patient with severe liver problems that their condition is benign and erroneously advised the use of a Pap smear to detect vaginal cancer.
The Department of Health and Human Services (HHS), meanwhile, has been using xAI’s Grok to provide dietary advice to Americans. “Use AI to get real answers about real food,” an official government website suggests. The page redirects users to the same AI system that has generated child sexual abuse images and praise for Hitler. The department’s use of Grok is part of a broader effort from Health Secretary Robert F. Kennedy Jr. to promote AI as transformational for “medicine in this country with its capacity to diagnose.”
“Costs and confusion”
As the U.S. faces a severe physician shortage, many people are seeking answers online. The results can be beneficial, with some chatbot users citing the technology as a source of critical health information. A pregnant woman in South Carolina told Newsweek last year that ChatGPT helped her identify a blood pressure issue, which then led her to seek emergency help for undiagnosed preeclampsia. Models from several AI companies have also scored highly on medical licensing exams.
However, a February study published in Nature Medicine that tested OpenAI’s GPT-4o, Meta’s Llama 3, and Cohere’s Command R found the models were less beneficial in real-world scenarios. They achieved an accuracy rate of just 35% correctly when presented with conversational descriptions of medical problems by a group of 1,300 volunteers who lacked medical expertise. The models ultimately proved about as beneficial as a standard internet search in helping patients diagnose their conditions.
Despite their shortcomings, it’s clear why so many Americans look to chatbots for treatment advice. Earlier this month, Kaiser reported that 2024 marked the first year since 2019 in which the uninsured rate increased in America. Things could get even worse in the coming months. At the end of last year, Congress allowed tax credits that significantly lowered premiums for millions of Affordable Care Act enrollees to expire. And millions more Americans are set to lose healthcare access as a result of Donald Trump’s One Big Beautiful Bill Act, which carved out tax cuts for the wealthy and corporations by slashing $1 trillion from Medicaid spending through 2034.
As it happens, Meta, Amazon, and Google-parent Alphabet, three of the companies now hawking AI-powered health advice to Americans, have all benefited from tax cuts that were partially offset by reductions in healthcare spending. The companies avoided a combined $49.6 billion in federal income taxes last year, according to the Institute on Taxation and Economic Policy.
Cuts to Medicaid, which will be felt in earnest beginning in 2027, are likely to push more Americans to lean on unreliable and unsafe chatbots for health advice. Results from the Kaiser survey published last month found that 20% of Americans who use AI for medical guidance said they did so because they lacked a health care provider or were unable to get a doctor’s appointment. Another 20% said they resorted to chatbots because they struggle to afford health care, a figure that jumped to 32% among Americans with annual incomes below $40,000. Black, Hispanic, and under-29 respondents were also more likely to use AI for physical and mental health advice.
In a press release announcing its Health AI agent, Amazon acknowledged the obstacles that prevent many Americans from receiving adequate health care. “Patients delay care because of costs and confusion,” wrote Amazon Health Services executive Prakash Bulusu and Amazon One Medical executive Andrew Diamond. “They face weeks-long waits for appointments, repetitive paperwork, hurried appointments at inconvenient times, impersonal disconnected systems and apps, and a maze of insurance rules.”
“We felt compelled to do something about it,” the executives added, “and that’s where Health AI comes in.”
GIGO. Garbage In, Garbage Out
I guess muskrat & bros feel they haven't scooped up enough private data on every single one of us yet. And none of their tech antics has taught them that one of humanity's (here in the US at least) problems is the lack of genuine human-to-human interaction, in every way, but especially when psychological or physical problems exist. (Our stats on depression, gun violence, and suicide aren't high enough yet???) Health professionals should know better as well. This stuff makes me sick.