Critical gaps in health technology assessment guidance for generative AI across six health systems

Abstract The integration of artificial intelligence (AI) in healthcare poses significant challenges for evaluation due to algorithmic complexity and evolving performance. Existing evidence generation guidelines, primarily designed for static medical technologies, may inadequately address the unique characteristics of generative AI. Here, we conducted a comparative analysis of evidence generation guidelines from six countries, examining recommendations for efficacy and effectiveness evaluations, health economic evaluations, social and equity evaluations, and uncertainty management for AI technologies. Across 56 documents, we found a consistent preference for randomised controlled trials and cost-effectiveness analyses, alongside emerging acceptance of alternative study designs and real-world evidence to accommodate the evolving nature of AI technologies. However, current frameworks lack specific guidance for the stochastic, non-deterministic, and adaptive learning properties of generative AI technologies. Moving forward, evidence generation guidelines require methodological and conceptual adaptations in order to improve the value assessment for generative AI technologies.