Creating a Large Language Model to Catalog Important Radiologist Recommendations

Medical errors, the third leading cause of death in the U.S., include wrong or delayed diagnoses, causing more serious harm than any other type of medical error. Delayed or missed opportunities for diagnoses (MOD) are particularly common in diagnostic imaging, where incidental findings often require further evaluation. At Parkland Health, a major safety-net public health system, 1.7 percent of all CT and MRI studies involve such findings. To address this, a large language model (LLM) is developed that identifies and flags delayed surveillance recommendations from radiologists’ interpretations. These delayed recommendations result in MODs 17 percent of the time. This LLM has been integrated into the electronic health record (EHR) of Parkland Health, enabling centralized management and navigation of these cases. Our results demonstrate 95 percent accuracy in identifying imaging that requires follow-up based on physician notes and 85 percent accuracy in determining the appropriate timing for follow-up. This work outlines the process, development, tools, current performance, and future plans for building an automated system to enhance image surveillance and mitigate MODs in diagnostic imaging.