Quality assessment of MR images: Does deep learning outperform machine learning with handcrafted features on new sites?

Motivation: Deep learning (DL) outperforms conventional machine learning (ML) that relies on handcrafted feature-based in many vision tasks, but its superiority in assessing brain MRI image quality for new sites/scanners is unclear. Goal(s): Compare DL and conventional ML for quality assessment of brain MRI images from new sites/scanners. Approach: One popular and widely accepted DL and one conventional ML method are evaluated on a multi-site dataset using leave-one-site-out approach using a binary quality label (good/bad). Results: Averaged balanced accuracy (BA) for the DL and conventional ML approaches are comparably poor (0.60+-0.12 and 0.54+-0.12, respectively) and does not exceed 0.76, suggesting room for improvement. Impact: Widespread adoption of automated quality assessment of brain MRI images is limited by a lack of generalizability. By comparing popular DL and conventional ML approaches, we find comparable but limited generalizability. This underscores the need for future algorithm development.