SMARTEN: a human-AI hybrid framework for assisted medical literature analysis and its evaluation

For medical professionals, the comprehension and analysis of literature is essential, however, it requires time which could be dedicated to the primary duties of providing care to patients. To address these conflicting needs and the challenge of finding new and appropriate information at the right time during medical literature review analysis, we introduce, to the best of our knowledge, the first automated framework co-designed with medical professionals, utilising a human-AI hybrid approach for the consolidation of large volumes of literature into manageable, topic-based chunks. For this framework (named SMARTEN), on the AI side, cluster-based topic modelling is deployed for the analysis of academic articles (consisting of abstracts and full article content) extracted from the PubMed database, the go-to venue for medical professionals, to ensure the provision of a wide range of topics reflecting this rich field and access to large volumes of biomedical literature. After consolidating topics within the literature, semantic analyses of the relationships between topics are performed, based on the content of the literature within each topic, ultimately assisting in the discovery of new knowledge from the literature. The effectiveness of the proposed approach was evaluated through a software implementation of the SMARTEN framework (the SMARTEN system), and a subsequent experimental study upon the concepts of cognitive load, technology acceptance, and general satisfaction by 18 medical practitioners and medical students. Moreover, the labels assigned by participants to publications were collected and analysed to explore the average accuracy for identifying relevant publications in the SMARTEN system. Overall, our analysis of user feedback and user logs indicates that the SMARTEN systsem contributed to enhancing the discovery of new knowledge within the literature, while maintaining cognitive load at manageable levels for medical professionals and students. Furthermore, our SMARTEN system effectively identifies relevant publications that would typically appear far down in PubMed search rankings, consistently positioning them within the top 10 results. This study presents the SMARTEN framework for the automated analysis of large volumes of academic literature based on the semantic contents of individual publications via cluster-based topic modelling. This enables the expediting of literature review for busy medical experts, while also enabling the discovery of new knowledge which would otherwise have gone unnoticed in the wealth of research available online. We evaluate the system developed based on this framework in an experimental study with medical experts, to reinforce the benefit that this framework provides and to identify improvements for further research.