Healthcare AI Challenge to Crowdsource Input from Clinicians
Mass General Brigham’s artificial intelligence group is hosting a Healthcare AI Challenge, a multi-institutional virtual, interactive series of events where healthcare professionals can assess the latest AI technologies in real-world scenarios.
Other participants in the Healthcare AI Challenge Collaborative include Emory Healthcare, the Department of Radiology at the University of Wisconsin School of Medicine and Public Health, and the Department of Radiology at the University of Washington School of Medicine. The American College of Radiology has also joined the collaborative.
“The velocity of AI innovations and breadth of their healthcare applications continues to increase. This unprecedented growth leaves clinicians struggling to determine the effectiveness of these innovations in safely delivering value to healthcare providers and our patients,” explained Keith Dreyer, D.O., Ph.D., chief data science officer at Boston-based Mass General Brigham, in a statement.
“The Healthcare AI Challenge is a collective response to the complexities involved in advancing the responsible development and use of AI in healthcare. This new approach strives to put clinicians in the driver’s seat, allowing them to evaluate the utility of different AI technologies and ultimately, determine which solutions have the greatest promise to advance patient care,” added Dreyer, leader of Mass General Brigham AI, the healthcare system’s AI business.
Participating healthcare professionals will be granted access to the Healthcare AI Challenge that features new AI solutions they can assess for effectiveness on specific medical tasks, such as providing medical image interpretation, in a simulated environment. Participants with relevant healthcare credentials can then provide their feedback on the solutions’ performance and utility, which will generate publicly available insights and analytics. By crowdsourcing input from healthcare professionals, the Healthcare AI Challenge seeks to create continuous, consistent and reliable expert evaluations of AI solutions in medicine. Importantly, scaling the evaluation of these technologies and sharing the insights broadly and transparently can result in societal benefit for healthcare stakeholders and patients globally.
“We need to go beyond collaboratives that come to consensus on how to think about AI,” said Alistair Erskine, M.D., chief information and digital officer at Emory Healthcare and Emory University, in a statement. “We need healthcare delivery communities to provide real-world experience of the application of AI at the point of care. That is what the Healthcare AI Challenge is designed to do.”
Radiology was chosen as the first in a series of healthcare AI events given the historic use of AI solutions to make rapid and meaningful impact in the field against the wave of generative AI model proliferation.
“Medical imaging provides many types of data, and up to 95% of healthcare data collected is unstructured, non-text data,” said Richard Bruce, M.D., associate professor of radiology and the vice chair of informatics at the University of Wisconsin School of Medicine and Public Health, in a statement. “AI has the potential to interpret and distill that data at a new scale and speed, but what we need is the ability to quickly test and compare different AI solutions. The Healthcare AI Challenge will offer a platform to evaluate and compare tools across various clinical situations.”
Healthcare professionals at institutions that are a part of the Healthcare AI Challenge Collaborative who register can log onto the Healthcare AI Challenge, select one of several events—such as image interpretation—and choose from a series of challenges to assess any one of the multiple foundation models available on the platform. The image interpretation challenges include questions focused on draft report generation, key findings, differential diagnosis, among others. The expert then rates the clinical skill level of the foundation models’ responses, which contributes to the insights and analytics rankings. Only verified healthcare professionals can participate in challenges that contribute to the rankings. The results of the Healthcare AI Challenge can be followed by the general public at HealthcareAIChallenge.org.
“We are facing an overwhelming influx of FDA-approved AI tools in healthcare, especially in radiology. Forming an academic collaborative could play a crucial role in validating and selecting these tools, ensuring they adhere to the highest standards of efficacy and safety,” said Dushyant Sahani, M.D., chair and professor of the Department of Radiology at the University of Washington School of Medicine, in a statement.
One goal is that the rankings generated by the Healthcare AI Challenge could serve to provide industry, healthcare stakeholders, and the public with a transparent analysis of AI solutions’ performance across a wide range of healthcare data and clinical scenarios.
User activity and expert feedback are expected to provide valuable insights to help AI developers better understand how healthcare professionals consider AI healthcare solutions in the context of providing clinical value. In turn, developers can enhance these technologies so that they are fit-for-purpose, commercially viable and clinically relevant. This may foster collaborations and knowledge-sharing with industry to help ensure responsible adoption of AI in healthcare settings.
