Intelligent mobile multimodal optical imaging and analysis framework using smartphone and deep feature fusion

We demonstrate an intelligent, smartphone-based multimodal optical imaging and analysis framework designed for accurate and mobile diagnosis of skin and ear diseases. Although early detection of these conditions is essential for effective treatment, it remains challenging due to overlapping visual symptoms and limited access to high-quality imaging technologies. The proposed framework includes a modular and scalable smartphone-based optical system that supports the integration of spectral, fluorescence, polarization, and 3D imaging modalities. This configuration enables the acquisition of rich structural and biochemical information from target regions in a compact, portable format. The system was applied to both dermatological and otological assessments. For skin disease diagnosis, clinical studies were conducted on patients with seborrheic dermatitis and psoriasis. Simultaneously, the system was evaluated for otological applications, particularly for distinguishing between various ear diseases, including otitis media with effusion. To improve diagnostic accuracy, we developed a deep learning framework that incorporates a CNN-based feature extractor and a Gated Feature Fusion (GFF) mechanism. This strategy dynamically adjusts the weight of each imaging modality based on its relevance to the target condition, enabling adaptive and context-aware feature integration. Meanwhile, to address the limited multimodal datasets, we employed an Auxiliary Classifier GAN (AC-GAN) to generate realistic synthetic images for model training. These augmented datasets were validated both qualitatively and quantitatively, resulting in improved diagnostic performance. In summary, our system combines hardware innovation with advanced AI-driven analysis, offering a cost-effective, portable, and scalable solution for early diagnosis and continuous monitoring of skin and ear diseases. It holds strong potential for intelligent, point-of-care diagnostics in resource-limited settings.