Advanced Transformer-Based System to Localize and Predict Atopic Dermatitis

Atopic Dermatitis is a chronic inflammatory skin disease that requires accurate severity assessment for effective management. The Eczema Area and Severity Index (EASI) is a widely used scoring system; however, traditional manual scoring is subjective and time-consuming. Previous works have attempted to automate this task using machine learning, such as methods relying on hand-engineered features, and more recently, EczemaNet, which introduced a two-stage scoring system utilizing deep convolutional neural networks (CNNs). Despite their advancements, these methods still have limitations when applied to clinical data. To address these, we propose improving the existing two-stage machine learning system by fine-tuning pre-trained Vision Transformer (ViT) and hybrid models that utilize both attention and convolution mechanisms. Leveraging ViT’s global context capture and hybrid models’ balance of local and global features, our system performs better than existing CNN-based systems on our internal clinical trial data, with a $5.72 \%$ increase in accuracy using ViT and a $2.69 \%$ increase in accuracy using hybrid models. We also compare several fine-tuning methods and investigate their tradeoffs between effectiveness and efficiency. Moreover, we explore the potential of using cutting-edge multi-modal models to generate descriptive diagnostic information, improving interpretability for clinicians and patients.