Deep multi-task learning framework for gastrointestinal lesion-aided diagnosis and severity estimation

Accurate diagnosis and severity estimation of gastrointestinal tract (GT) lesions are crucial for patient care and effective treatment plan decisions. Traditional methods for diagnosing lesions face challenges in accurately estimating severity due to requiring interpretable biomarkers, inter-observer variability, and overlapping lesions. Moreover, existing deep-learning models treat lesion classification and severity estimation as separate tasks, complicating diagnosis. To address these gaps, we propose a deep multi-task learning framework that aims to improve accuracy by simultaneously addressing classification and severity estimation. The proposed framework is designed in three stages utilizing four multi-class GT datasets. The first stage involves multi-scale feature representation using the convolutional vision transformer (CViT) blocks. The CViT with enhanced multi-head attention employs a deep multi-task learning approach extracting shared features in a unified manner. In the second stage, the extracted features are combined with the features from the first stage. In a subsequent stage, task-specific enhanced multi-head attentions are applied to the concatenated features to facilitate efficient learning between global and local information features. Our approach enhances fine-grained image features by incorporating semantic image features and focusing on representation subspace. Extensive experimental results demonstrate significant performance, validating the proposed model's effectiveness across various datasets in lesion diagnosis and severity estimation.