Validation of the QAMAI tool in Italian for the evaluation AI-generated health information in head and neck surgery

BACKGROUND: This study aimed to validate the Italian version of the Quality Assessment of Medical Artificial Intelligence (IT-QAMAI) tool, designed to evaluate the reliability of AI-generated health information in the context of head and neck surgery.METHODS: The IT-QAMAI tool was adapted from the original English version and involved a rigorous translation and back-translation process. The validation involved 18 researchers from 13 centers across Europe, assessing 24 AI-generated responses categorized into clinical scenarios, theoretical questions, and patient inquiries. The tool’s reliability was measured using Cronbach’s alpha for internal consistency, the Intraclass Correlation Coefficient (ICC) for inter-rater reliability, and Pearson’s correlation for test-retest reliability.RESULTS: The IT-QAMAI demonstrated high internal consistency (Cronbach’s alpha = 0.850) and good inter-rater reliability (ICC=0.750). Test-retest reliability was strong (rs=0.887). Significant differences were found in the quality of AI-generated responses across different question types.CONCLUSIONS: The IT-QAMAI tool is a reliable and valid instrument for assessing the quality of AI-generated health information in Italian, with significant implications for its use in clinical practice and research in head and neck surgery.