The Risks of Generative AI
Generative AI: A Risky Proposition for Healthcare
Generative AI, while powerful, has shown significant shortcomings in healthcare applications. A 2024 study by the University of Massachusetts Amherst and Mendel uncovered alarming rates of hallucinations in medical summaries generated by frontier AI models.
Key Findings:
- Nearly all summaries analyzed contained hallucinations.
- Inconsistencies included:
- 327 medical event errors in GPT-4o summaries.
- 271 medical event errors in Llama-3 summaries.
- Incorrect reasoning and chronological inconsistencies were common.
- The most frequent hallucinations were related to symptoms, diagnoses, and medicinal instructions, posing serious risks such as misdiagnosis or incorrect treatment.
