Implementing a Commercial AI Fracture Detection Tool in Health Care Using the Non-Adoption, Abandonment, Scale-Up, Spread, and Sustainability Framework: A Formative Evaluation Study

Background: Artificial intelligence (AI) has the potential to enhance resource efficiency, improve patient treatment, and increase safety in health care. Still, there is limited knowledge on how to implement and evaluate AI solutions in real-world clinical settings. To address this gap, we conducted a formative process evaluation of the first large-scale procurement and implementation of a commercial AI solution in Norwegian health care. F The Non-Adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework, was used for the formative process evaluation throughout the 4-year project to guide data collection, analysis, and real-time feedback. Objective: This study aimed to evaluate the usefulness of the NASSS framework for formative process evaluation of AI implementation in health care. Methods: A formative process evaluation was conducted from 2020 to 2024, covering the procurement, preimplementation, and implementation phases. Data included 65 interviews, observations, and document analysis. Data were analyzed thematically using the 7 NASSS domains, supplemented with subtopics within each domain to capture emerging infrastructural complexities and temporal dynamics. Real-time findings were discussed with the implementation team, decision-makers, and clinicians. Results: Key factors for successful implementation included clinician trust, workflow integration, task distribution, and digital maturity. Major challenges comprised limited documentation of Conformité Européenne-marked solutions, deskilling, and misaligned financial incentives. The NASSS framework enabled the identification of sociotechnical values and complexities, but did not fully capture workflow evolution and changing user perceptions over time. Conclusions: The NASSS framework is useful for evaluating AI implementation but requires adaptation to capture temporal dynamics and workflow changes better. These findings contribute to improving evaluation approaches for AI in health care.