Researchers: LLMs Can Help Process Hospital Quality Measures

Researchers at University of California San Diego School of Medicine found that large language models (LLMs) can accurately process hospital quality measures, achieving 90% agreement with manual reporting.

By addressing the complex demands of quality measurement, the researchers believe the findings pave the way for more efficient and reliable approaches to healthcare quality reporting.

The results of the pilot study were published in the Oct. 21, 2024 online edition of the New England Journal of Medicine (NEJM) AI. 

Researchers of the study, in partnership with the Joan and Irwin Jacobs Center for Health Innovation at UC San Diego Health (JCHI), found that LLMs can perform accurate abstractions for complex quality measures, particularly in the context of the Centers for Medicare & Medicaid Services (CMS) SEP-1 measure for severe sepsis and septic shock.

Traditionally, the abstraction process for SEP-1 involves a meticulous 63-step evaluation of extensive patient charts, requiring weeks of effort from multiple reviewers. This study found that LLMs can dramatically reduce the time and resources needed for this process by accurately scanning patient charts and generating crucial contextual insights in seconds.

“The integration of LLMs into hospital workflows holds the promise of transforming healthcare delivery by making the process more real-time, which can enhance personalized care and improve patient access to quality data,” said Aaron Boussina, postdoctoral scholar and lead author of the study at UC San Diego School of Medicine, in a statement. “As we advance this research, we envision a future where quality reporting is not just efficient but also improves the overall patient experience.”

Boussina is a co-founder of and holds equity in Healcisio Inc., a start-up that develops products related to digital health. This study was funded, in part, by a monetary award provided to Healcisio in which University of California San Diego was a sub-recipient. 

The study also found that LLMs can improve efficiency by correcting errors and speeding up processing time; lowering administrative costs by automating tasks; enabling near-real-time quality assessments; and are scalable across various healthcare settings.

Future steps include the research team validating these findings and implementing them to enhance reliable data and reporting methods. 

“We remain diligent on our path to leverage technologies to help reduce the administrative burden of health care and, in turn, enable our quality improvement specialists to spend more time supporting the exceptional care our medical teams provide,” said Chad VanDenBerg, M.P.H., study co-author and chief quality and patient safety officer at UC San Diego Health, in a statement. 

The study was funded, in part, by the National Institute of Allergy and Infectious Diseases, the National Library of Medicine and the National Institute of General Medical Sciences.