Artificial intelligence (AI) holds promise to enable new approaches to diagnostic testing and digital pathology; however, realizing this potential requires coordinated approaches to evaluate performance and support consistency across platforms. A new manuscript from Friends of Cancer Research (Friends) Digital PATH Project, published in Modern Pathology, establishes a foundation for advancing conversations around evaluating performance and variability across novel AI tools to build trust, ensure consistency, and unlock the full potential of AI applications in advancing oncology drug development and care.

Read the Publication

Ten independently developed computational pathology AI models scored 1,124 whole slide images from 733 breast cancer patients, assigning scores from 0 to 3+ based on HER2 positivity. The authors assessed agreement across AI tools and against human readers. Agreement was significantly higher for HER2-high cases, with AI models achieving 97% consensus on HER2 3+ assessments. This approach provided a practical method to evaluate AI tools and characterize agreement across independently developed models.

“Differences in model design—such as how HER2 staining is interpreted or how tumor regions are defined—likely contribute to variability, underscoring the need for transparent and standardized performance evaluation” said Dr. Roberto Salgado, ZAS Hospital oncologist and Digital PATH working group member.

Heatmap showing how different AI models (and human pathologists) scored the same breast cancer tissue samples for HER2. While the models and pathologists aligned well on HER2 3+ (dark blue), there was more variability in HER2 1+ and 2+ (light green) cases.

Nearly all patients diagnosed with breast cancer undergo testing to assess HER2 levels, a key biomarker used to guide treatment decisions. Its clinical significance and the availability of robust datasets made it an ideal case study for the project. These findings provide a framework for evaluating AI variability that could be applicable to other AI-enabled biomarker assessments.

This project highlights how an independent set of images can be used to demonstrate model performance and generate insights that help inform future regulatory policy considerations. Identifying approaches for creating standardized reference datasets to support tool accuracy and consistency will be important as the field of digital pathology continues to evolve.

The Digital PATH project is part of the Friends Diagnostics Harmonization Portfolio that focuses on standardizing diagnostics tests by generating evidence to characterize variability and support accuracy and consistency across different testing methods for patients.

Read the publication: https://doi.org/10.1016/j.modpat.2025.100944.

To learn more about the Digital PATH Project, please visit: https://friendsofcancerresearch.org/digital-pathology/.

Authors
Brittany McKelvey¹, Pedro A. Torres-Saavedra², Jessica Li², Glenn Broeckx³, Frederik Deman³, Siraj Ali⁴, Hillary Andrews¹, Salim Arslan⁵, Meir Azulay⁶, Santhosh Balasubramanian⁷, J. Carl Barrett⁸, Peter Caie⁹, Ming Chen¹⁰, Daniel Cohen¹¹, Tathagata Dasgupta¹², Diana Fahrer⁹, George Green¹³, Mark Gustavson¹⁴, Sarah Hersey¹⁵, Ana Hidalgo-Sastre¹⁶, Shahanawaz Jiwani¹⁷, Elaine Joseph¹⁸, Wonkyung Jung⁴, Kimary Kulig¹⁹, Vladimir Kushnarev²⁰, Jochen Lennerz²⁰, Xiaoxian Li²¹, Meredith Lodge⁹, Joan Mancuso²², Mike Montalto²³, Satabhisa Mukhopadhyay¹², Foivos Ntelemis⁵, Matthew Oberley¹⁰, Pahini Pandya⁵, Oscar Puig⁶, Edward T. Richardson²⁴, Alexander Sarachakov²⁰, Mark Stewart¹, Lisa M. McShane², Roberto Salgado²⁵, Jeff Allen¹

Project Partners
¹Friends of Cancer Research, ²Division of Cancer Treatment and Diagnosis, National Cancer Institute, ³PA², Dept. of Pathology, Ziekenhuis aan de Stroom (ZAS), Antwerp, Belgium and Centre for Oncological Research (CORE), MIPPRO, Faculty of Medicine, Antwerp University, ⁴Lunit, ⁵Panakeia Technologies, ⁶Nucleai, ⁷PathAI, ⁸University of North Carolina at Chapel Hill, ⁹Indica Labs, ¹⁰Caris Life Sciences, ¹¹GSK, ¹²4D Path, ¹³GA Green Consulting LLC, ¹⁴AstraZeneca, ¹⁵Bristol Myers Squibb, 1¹⁶AstraZeneca Computational Pathology GmbH, ¹⁷Molecular Characterization Laboratory, Frederick National Laboratory/National Cancer Institute, ¹⁸AstraZeneca, ¹⁹Kulig Consulting, ²⁰BostonGene, ²¹Emory University, ²²Patient Advocate, ²³Amgen,²⁴Merck & Co., Inc., ²⁵PA², Dept. of Pathology, Ziekenhuis aan de Stroom (ZAS), Antwerp, Belgium and Division of Research, Peter Mac Callum Cancer Centre