Artificial Intelligence Model’s assessment of intra-sample heterogeneity of HER2 IHC in breast cancer is related to interobserver and intraobserver agreement among pathologists

Accurate assessment of HER2 status in breast cancer has been critical for guiding therapy and has become even more important with the emergence of antibody–drug conjugates, now also indicated in HER2-low tumors. However, inter- and intraobserver variability limits the reproducibility of HER2 IHC scoring among pathologists. Artificial intelligence (AI) models offer potential to standardize and improve diagnostic accuracy and bring new insights into current practices shortcomings. We conducted a study recruiting generalist and specialist pathologists from Rede D’Or centers across Brazil to assess digitized HER2 IHC whole slide images. The same images were presented for the pathologists with an interval of one month and to the AIM-HER2 (PathAI ®, Boston, MA) AI model. Intra- and interobserver agreement, as well as agreement with AI, were measured across 126 breast cancer samples. The association between sample features and agreement metrics was also analyzed using AI spatial breakdown data. Among pathologists, the median intraobserver agreement was 66.67%, and the median agreement with AI was 60.8%. Median interobserver agreement was 67.65%, with high agreement (> 85%) in 25.4% of samples. Significant positive correlations were observed among all agreement metrics. Samples with lower intra-sample heterogeneity, as determined by AI spatial breakdown scores, were associated with higher agreement levels. Our findings highlight significant variability in HER2 IHC scoring among pathologists. We found that AI-assessed intra-sample heterogeneity is correlated with a lower agreement rate among pathologists. We believe this shows a potential limitation of current scoring practices and that AI models such as AIM-HER2 can have a role in increasing reproducibility of analysis by indicating the samples that are more likely to result in diagnostic discordance between evaluators.