New Collaborative Will Publicly Rank Top Health Care AI Applications

Amid the explosive growth in generative artificial intelligence (AI) applications in health care, clinicians often have struggled to determine the effectiveness of these tools in safely delivering value to the care process.

A new collaboration led by Mass General Brigham will provide a multi-institutional virtual, interactive series of events in which professionals can explore and assess the latest AI health care technologies in real-world scenarios.

The so-called Healthcare AI Challenge also includes health care professionals from Emory Healthcare; the department of radiology at the University of Wisconsin School of Medicine and Public Health; and the department of radiology at the University of Washington School of Medicine. The American College of Radiology, a professional medical society representing radiologists, also joined the Healthcare AI Challenge collaborative as a founding member to ensure that its 42,000-member community has access to it.

Participating health care professionals will be granted access to the Healthcare AI Challenge, which features late-breaking AI solutions they can assess for effectiveness on specific medical tasks, such as providing medical image interpretation, in a simulated environment. Participants with relevant health care credentials then can provide their feedback on the solutions’ performance and utility, which will generate publicly available insights and analytics.

By crowdsourcing input from health care professionals, the Healthcare AI Challenge seeks to create continuous, consistent and reliable expert evaluations of AI solutions in medicine. Scaling the evaluation of these technologies and sharing the insights broadly and transparently are expected to provide benefits for health care stakeholders and patients globally.

Health care professionals at institutions that are part of the collaborative and who register can log onto the Healthcare AI Challenge, select one of several events — such as image interpretation — and choose from a series of challenges to assess any of the multiple foundation models available on the platform.

The image interpretation challenges include questions focused on draft report generation, key findings and differential diagnosis, among others. The expert then rates the clinical skill level of the foundation models’ responses, which contributes to the insights and analytics rankings. Only verified health care professionals can participate in challenges that contribute to the rankings. The results of the Healthcare AI Challenge can be followed by the general public at HealthcareAIChallenge.org.