Rethinking Rigor: Towards a Consortium of Benchmarking Initiatives to Validate AI Models in Biomedicine

The explosive growth of data-driven artificial intelligence in biomedicine is reshaping the scientific landscape, powering innovative healthcare solutions and transformative insights in life sciences. AI is proving to be a new paradigm for predictive modeling, which in turn demands a paradigm shift in our approach to model evaluation. Traditional algorithmic performance metrics alone are insufficient for comprehensive validation. Concurrent with the rise of AI modeling opportunities has been a proliferation of community-led benchmarking initiatives, fragmenting efforts and limiting their collective synergy and impact. This talk proposes fostering a coalition of Critical Assessment efforts. Such a coalition can enhance collaboration, establish rigorous validation standards, and build sustained trust in evolving AI methodologies, ensuring their robust and reliable integration into biomedical research.