Automating in High-Expertise, Low-Label Environments: Evidence-Based Medicine by Expert-Augmented Few-Shot Learning

Many real-world process automation environments are rife with high-expertise and limited labeled data. We propose a computational design science artifact to automate systematic review (SR) in such an environment. SR is a manual process that collects and synthesizes data from medical literature to inform medical decisions and improve clinical practice. Existing machine learning solutions for SR automation suffer from a lack of labeled data and a misrepresentation of the high-expertise manual process. Motivated by humans’ impressive capability to learn from limited examples, we propose a principled and generalizable few-shot learning framework—FastSR—to automate the multistep, expertise-intensive SR process using minimal training data. Informed by SR experts’ annotation logic, FastSR extends the traditional few-shot learning framework by including (1) various representations to account for diverse SR knowledge, (2) attention mechanisms to reflect semantic correspondence of medical text fragments, and (3) shared representations to jointly learn interrelated tasks (i.e., sentence classification and sequence tagging). We instantiated and evaluated FastSR on three test beds: full-text articles from Wilson disease (WD) and COVID-19, as well as a public dataset (EBM-NLP) containing clinical trial abstracts on a wide range of diseases. Our experiments demonstrate that FastSR significantly outperforms several benchmarking solutions and expedites the SR project by up to 65%. We critically examine the SR outcomes and practical advantages of FastSR compared to other ML and manual SR solutions and propose a new FastSR-augmented protocol. Overall, our multifaceted evaluation quantitatively and qualitatively underscores the efficacy and applicability of FastSR in expediting SR. Our results have important implications for designing computational artifacts for automating/augmenting processes in high-expertise, low-label environments.