Generative AI, the can create and analyze images, text, audio, videos and more, is increasingly finding its way into healthcare and is being pushed forward by both Big Tech companies and start-ups.
Google Cloud, Google’s cloud services and products division, is collaborating with Highmark Health, a Pittsburgh-based nonprofit healthcare company, on generative AI tools to personalize the patient intake experience. Amazon’s AWS division says it is working with unnamed customers on a way to use generative AI for analytics medical databases for “social determinants of health.” And Microsoft Azure is helping build a generative AI system for Providence, the nonprofit healthcare network, to automatically triage messages sent by patients to providers.
Prominent healthcare generative AI startups include Ambience Healthcare, which is developing a generative AI app for doctors; Nabla, an ambient AI assistant for practitioners; and Abridge, which develops medical documentation analysis tools.
The widespread enthusiasm for generative AI is reflected in the investments in generative AI efforts in healthcare. Overall, generative AI in healthcare startups have raised tens of millions of dollars in venture capital to date, and the vast majority of healthcare investors report that generative AI has significantly influenced their investment strategies.
But both experts and patients are divided over whether healthcare-focused generative AI is ready for prime time.
Generative AI may not be what people want
In a recent Deloitte survey, only about half (53%) of U.S. consumers said they believe generative AI could improve healthcare—for example, by improving access or reducing wait times for appointments. Less than half said they expected generative AI to make medical care more affordable.
Andrew Borkowski, chief AI officer at VA Sunshine Healthcare Network, the U.S. Department of Veterans Affairs’ largest health system, doesn’t think the cynicism is unwarranted. Borkowski warned that the use of generative AI may be premature due to its “significant” limitations – and concerns about its effectiveness.
“One of the main problems with generative AI is its inability to handle complex medical questions or emergencies,” he told TechCrunch. “Its limited knowledge base – that is, lack of current clinical information – and lack of human expertise make it unsuitable for providing comprehensive medical advice or treatment recommendations.”
Several studies suggest that these points are credible.
An article in the journal JAMA Pediatrics found that OpenAI’s generative AI chatbot ChatGPT, which some healthcare organizations have tested for limited use cases, makes errors 83% of the time when diagnosing pediatric diseases. And when testing OpenAI’s GPT-4 as a diagnostic assistant, doctors at Beth Israel Deaconess Medical Center in Boston found that the model ranked the wrong diagnosis as the top answer nearly two out of three times.
Today’s generative AI also struggles with medical administrative tasks that are an integral part of doctors’ daily workflows. In the MedAlign benchmark, to evaluate how well generative AI can do things like summarize patient records and search notes, GPT-4 failed 35% of the time.
OpenAI and many other generative AI providers warn against relying on their models for medical advice. But Borkowski and others say they could do more. “Relying solely on generative AI in healthcare could lead to misdiagnosis, inappropriate treatment or even life-threatening situations,” Borkowski said.
Jan Egger, who leads AI-driven therapies at the Institute for AI in Medicine at the University of Duisburg-Essen, which studies the applications of new technologies to patient care, shares Borkowski’s concerns. He believes that the only safe way to use generative AI in healthcare right now is under the strict, watchful supervision of a doctor.
“The results can be completely wrong and it is becoming increasingly difficult to maintain awareness of this,” Egger said. “Of course, generative AI can be used, for example, to pre-write discharge letters. But it is the responsibility of the doctors to check and make the final decision.”
Generative AI can perpetuate stereotypes
One particularly damaging way generative AI can go wrong in healthcare is by perpetuating stereotypes.
In a 2023 Stanford Medicine study, a team of researchers tested ChatGPT and other generative AI-powered chatbots on questions about kidney function, lung capacity and skin thickness. The co-authors found that not only were ChatGPT’s responses frequently false, but the responses also contained several long-reinforced, untrue assumptions that there are biological differences between blacks and whites – untruths that have been known to cause… medical professionals misdiagnosed health problems.
The irony is that the patients most likely to be discriminated against by generative AI in healthcare are also the ones most likely to use it.
People without health insurance — largely people of color, according to a KFF study — are more willing to try generative AI for things like finding a doctor or providing mental health support, according to the Deloitte survey. If AI recommendations are affected by bias, this could exacerbate disparities in treatment.
However, some experts argue that generative AI is making progress in this regard.
In a Microsoft study published in late 2023, researchers said they achieved 90.2% accuracy using GPT-4 on four demanding medical benchmarks. Vanilla GPT-4 was unable to achieve this score. However, the researchers say that through prompt engineering – designing prompts for GPT-4 to produce specific outputs – they were able to increase the model’s score by up to 16.2 percentage points. (It’s worth noting that Microsoft is a major investor in OpenAI.)
Beyond chatbots
But asking a chatbot a question isn’t the only thing generative AI is good for. Some researchers say medical imaging could benefit greatly from the power of generative AI.
In July, a group of scientists presented a system called cComplementarity-driven clinical workflow shift (CoDoC) in a study published in Nature. The system aims to figure out when medical imaging specialists should rely on AI instead of traditional techniques for diagnosis. According to the co-authors, CoDoC performed better than specialists while reducing clinical workflow by 66%.
In November, a Chinese research team demonstrated panda, an AI model for detecting potential pancreatic lesions in X-ray images. A study showed that Panda is very accurate in classifying these lesions, which are often detected too late for surgical intervention.
In fact, Arun Thirunavukarasu, a clinical research fellow at the University of Oxford, said there is “nothing unique” about generative AI that precludes its use in healthcare.
“More mundane applications of generative AI technology are feasible In “The solutions are available in the short and medium term and include text corrections, automatic documentation of notes and letters, and improved search functions to optimize electronic health records,” he said. “There is no reason why generative AI technology – if effective – cannot be used In I will take on such roles immediately.”
“Strict science”
But while generative AI shows promise in certain, limited areas of medicine, experts like Borkowski point to the technical and compliance hurdles that must be overcome before generative AI can be useful – and trustworthy – as a comprehensive assistive healthcare tool.
“Significant privacy and security concerns surround the use of generative AI in healthcare,” said Borkowski. “The sensitivity of medical data and the potential for misuse or unauthorized access pose significant risks to patient confidentiality and trust in the healthcare system.” Additionally, the regulatory and legal landscape surrounding the use of generative AI in healthcare is still evolving , although questions regarding liability, data protection and the practice of medicine by non-human entities still need to be resolved.”
Even Thirunavukarasu, who is optimistic about generative AI in healthcare, says there must be “rigorous science” behind patient-focused tools.
“Especially without direct oversight from physicians, there should be pragmatic randomized control trials demonstrating clinical benefit to justify the use of patient-focused generative AI,” he said. “Proper governance is critical going forward to guard against unexpected damage following a large-scale deployment.”
Recently, the World Health Organization published guidelines advocating for this type of science and human oversight of generative AI in healthcare, as well as the introduction of independent third-party audits, transparency and impact assessments of this AI. The aim, as outlined by the WHO in its guidelines, is to encourage a diverse cohort of people to participate in the development of generative AI for healthcare and to provide an opportunity to raise concerns and provide input throughout the process.
“Unless concerns are adequately addressed and appropriate safeguards are put in place,” Borkowski said, “widespread implementation of medical generative AI … could be potentially harmful to patients and the healthcare industry as a whole.”