AI07 Evaluating the quality of artificial intelligence-generated research: a comparison of Elicit and systematic reviews in dermatology

Abstract Artificial intelligence (AI) is increasingly being used in dermatology, where its application in diagnosing clinical images and histopathology has been growing. Its use in medicine has spread into optimizing administrative tasks and as a research aid. There have been concerns raised about fabricated studies being referenced by AI tools, prompting the development of dedicated programs. Elicit is one of the largest AI research tools. It can quickly summarize research based on clinical question prompts. While systematic reviews are considered the gold standard for evidence synthesis, evaluating AI tools like Elicit can help determine their potential to streamline research processes. This study aims to assess the quality of the studies produced by Elicit compared with systematic reviews in dermatology. Systematic reviews from the British Journal of Dermatology and reviews from the Journal of the American Academy of Dermatology were selected from the last 4 months of 2024. An Elicit search was generated based on the title of the systematic review. The first eight studies produced were assessed for inclusion in the systematic review and journal metrics. Twenty-two systematic reviews were included in the study. On average, 2.5 of the eight studies generated by Elicit were included in the corresponding systematic review. This was similar whether the systematic review included more than 10 studies or fewer (2.9 vs. 1.4). In total 69%of the studies generated by Elicit came from a journal with an impact factor > 2. The Elicit studies had similar journal metrics to the systematic reviews. Overall, 80% had impact factors within the same range as the review articles and 45% within the same year range. In six cases (27%), Elicit generated studies that were published later than those in the corresponding systematic reviews, potentially offering more up-to-date references. The Elicit search was generated based on the systematic review title to reduce bias. This limited the specificity of the search as relevant inclusion criteria were unable to be incorporated. This may have affected the concordance between AI-generated results and systematic review content. Overall, the AI research tool Elicit demonstrates the potential to produce reputable and relevant studies in dermatology, supporting its use as a research aid. Its limitations highlight the continued need for human oversight in evidence synthesis.