Consistent, accurate diagnoses are the bedrock of precision oncology. The nonprofit Friends of Cancer Research (Friends) held a meeting in Washington, DC, as part of its ongoing work to understand diagnostic test variability in support of improving precision medicine.
Invited speakers and panelists probed opportunities and challenges in the use of artificial intelligence (AI), alignment of frameworks for AI-enabled tools, companion diagnostics for rare biomarkers, and emerging policies for incorporating AI into healthcare. Friends released two white papers as background in advance of the meeting.
“In these times, we need to support science and innovation more than ever,” said Ellen Sigal, PhD, Friends Chair and Founder. In February 2024, the organization launched the Digital PATH Project, which is evaluating the variability between AI-enabled digital pathology tools using a common dataset. Digital PATH aims to help define future policies and advance the use of AI in diagnostic testing.
Examining Breast Cancer
As part of Digital PATH, Friends convened a working group to evaluate the HER2 biomarker in more than 1,100 whole slide images of breast cancer tissue across 10 computational platforms to better understand the factors that affect variability in biomarker assessment.
“AI tools may help address challenges in reproducibility, accuracy, and scalability in HER2 scoring,” said Mark Stewart, PhD, Friends Vice President for Science Policy.
In the first meeting session, Stewart presented data demonstrating variability across the models and pathologists studied by the working group using a scoring system on a scale of 0 to 3+. Possible reasons included sample type, artifacts, sample quality, storage, cellular heterogeneity, inadequate tumor cellularity, and heterogeneous staining patterns. The data showed there was greater consistency and concordance in HER2 3+ samples with less concordance in negative (1+) and borderline (2+) samples. Stewart said the next steps will include how to leverage these results to propose best practices with validated AI models for clinical use.
“Pathology is intrinsically hard,” said panelist Santhosh Balasubramanian, BSE, Director of AI Products at PathAI. He said the Friends Digital PATH Project will lead to best practices in designing reference datasets to validate AI tools to improve diagnosis, given the variability across AI models. What is exciting to him is the concept of a pathologist and an AI tool working as a team. There are certain things a pathologist will be better at and certain things an AI tool will be better at, and together they can improve diagnostic accuracy, Balasubramanian stressed.
Stopping Discrepancies
Panelist and breast cancer survivor Joan Mancuso, a Friends advisory advocate, knows personally how there can be discrepancies in diagnoses. On one test, she was diagnosed as HER2-positive (3+), but on another, she was diagnosed as HER2 borderline. In addition to careful use of AI, Mancuso said it is important for pathologists to consider patient characteristics, such as age, menopausal status, and ethnicity when making a diagnosis.
“AI will likely not replace a pathologist or diagnostic modality overall, but it will supplement it and I think that’s an incredible opportunity to have our experience plus a validated AI tool to ultimately render a more precise and more useful diagnostic decision,” said Joe Lennerz, MD, PhD, Chief Scientific Officer at BostonGene.
Panelists in this session agreed that strategic collaboration is vital for advancing AI-enabled pathology tools to enhance the accuracy and reproducibility of digital/computational methodologies. They also said publicly available reference datasets are needed to validate AI models.
Rare Biomarkers
In the second session of the Friends meeting, panelists probed regulatory processes of companion diagnostic tests for rare biomarkers, which affect 1 percent or less of those with a specific cancer.
For rare biomarkers, clinical samples are often limited, said Megan Doyle, JD, MPH, Associate Vice President and Assistant General Counsel-Diagnostics at Eli Lilly & Co. Therefore, she noted it can be advisable to use some alternative data sources, such as real-world evidence, for validation of a diagnostic test.
“We have to be really thoughtful about what’s best for patients as we’re doing these co-development programs,” Doyle said. “How do we get therapies that are safe and effective to patients as soon as practicable?”
The Friends white paper prepared for this session states: “Regulatory flexibilities can aid in demonstrating a favorable benefit-risk profile for rare biomarkers and indications, especially where there are limited clinical trial samples for validation studies.”
“We’ve witnessed some amazing strides” in identifying smaller biomarker subpopulations, said Elizabeth Mansfield, PhD, Vice President for Regulatory Affairs at Foundation Medicine, Inc. She noted that the FDA has been somewhat more open to the use of alternative data sources in recent years. “The goal is to get the best test out there in a timely manner,” Mansfield said, noting that what is needed for regulatory approval is a prospective biomarker plan, especially for a rare biomarker.
AI Diagnostics
In the final session at the Friends meeting, panelists probed the need for evolving AI governance and regulatory adaptation to address the iterative nature of AI-driven diagnostics, while ensuring patient safety and clinical utility. “AI is not a static tool; it is not a slide rule,” said panelist Joe Franklin, JD, PhD, Special Counsel at Covington & Burling, LLP. While regulatory flexibility is needed for the fast-paced field of AI-driven tools in healthcare, he noted that, in general, institutions are slow in incorporating computer-driven innovations into their daily workflow.
“There’s always that tension” between patient safety and advancing tools that can lead to cures and better care, said panelist Rohini Kosoglu, BS, Venture Partner at the Fusion Fund and former White House and Senate staffer. Agreeing with Franklin, she said the field of AI-driven diagnostics is moving very swiftly. She predicted that, in the future, it is likely that state legislatures will examine the use of AI tools in the context of consumer protection.
Panelists at this session raised the following considerations for discussion.
- What regulatory flexibilities are needed to help AI-driven diagnostics evolve? Franklin said the FDA’s recent guidance documents have been helpful in this regard in terms of establishing clearer regulatory parameters while also supporting the flexibility needed for innovation.
- How can a validated AI tool build trust and best fit into the existing, established workflow of an institution?
- How can the tension between patient safety and the need to incorporate validated new AI tools into best clinical practices be successfully managed?
- Given the iterative characteristics of AI-driven tools, what strategies are needed to re-evaluate the payer standards in clinical practice for reimbursement by the Centers for Medicare and Medicaid Services and private payers?
- In the future, what will the use of AI tools mean for patient consent forms and potential liability?
