Penn Researchers Seek to Develop More Accurate, Transparent AI Systems
Researchers at Penn Engineering and the University of Pennsylvania Perelman School of Medicine (PSOM) are working to develop artificial intelligence (AI) systems that support more personalized medical treatment.
The Penn researchers have received a $7 million, four-year award from the federal Advanced Research Projects Agency for Health (ARPA-H) to focus on breast cancer, heart attacks and sepsis initially.
The project will be led by Rajeev Alur, Zisman Family Professor in Computer and Information Science (CIS) at Penn Engineering and Director of the AI-enabled Systems: Safe, Explainable, and Trustworthy (ASSET) Center.
The projects will develop systems that help clinicians predict treatment response, leading to better health outcomes for patients. “We are excited to take on the challenge of building AI models for clinical predictions that are not only highly accurate but also accompanied with explanations that clinicians can trust,” said Alur in a statement.
Although machine learning and large language models have shown tremendous promise in classifying images and making inferences, two crucial tasks in medicine, those systems are not yet transparent nor accurate enough for clinical use, the researchers believe.
In part, this is because AI systems require vast amounts of data to make accurate predictions; since medical data is protected, AI systems must be specially trained for medical use. At present, AI systems also function like a “black box,” meaning that the steps that lead to their outputs cannot always be determined, leaving their “thought process” opaque.
To develop AI systems that are sufficiently accurate and transparent for medical use, the researchers at Penn Engineering and PSOM will explore multiple avenues within AI research, complementing data-driven models with logical and symbolic reasoning.
Penn said the researchers will work closely with clinicians to better understand how real-world constraints — such as when and how much data is available, how existing workflows can incorporate AI-powered support systems, and how the expected thresholds for various medical markers vary from patient to patient — can be incorporated into the system. Ultimately, the researchers’ goal is to support clinicians with predicting patients’ response to different treatments for breast cancer, heart attacks and sepsis, leading to better health outcomes.
Alur points to the partnership between Penn Engineering and PSOM as a catalyst for innovation. “The proposed research requires close collaboration among Penn Engineering faculty with expertise in machine learning and PSOM faculty with clinical expertise,” said Alur, in a statement “Recent initiatives within Penn Engineering and PSOM — from seed grants for AI researchers to our symposium on trustworthy AI — demonstrate how collaboration advances the field as a whole.”
“This is an exciting project that will bridge our expertise in AI with the clinical expertise of one of the world’s great health systems,” says Vijay Kumar, Nemirovsky Family Dean of Penn Engineering, in a statement. “Clinical challenges present so many important and impactful use cases for AI and Professor Alur has built a dream team to develop the foundations for trustworthy AI in medicine.”
