Interactive image diagnosis using test-time backpropagation from the auxiliary task

Multi-task deep learning has been developed to ensure explainability in computer-aided diagnosis (CAD) systems. These systems simultaneously predict interpretable characteristics of findings in auxiliary tasks to help radiologists assess the reliability of the diagnostic prediction. However, the CAD’s opinion would not be referred to when the auxiliary task prediction deviates from the radiologist’s expectation. To further leverage CAD’s knowledge in the above situation, we proposed a multi-task gradient assist (MGA) method that brings interactivity to multi-task models without additional training. MGA allows radiologists to give feedback to the CAD and ask for additional diagnostic suggestions. Using two datasets of mammography and CT, we show that MGA successfully improves diagnostic accuracy upon feedback on various auxiliary tasks. Compared to an existing method using forward propagation only, MGA performs more robustly and can be combined to further improve accuracy. Our method paves a promising way for integrating artificial intelligence into image diagnosis.