Multimodal Neural Network Analysis of Raman Spectra and Dermoscopic Images of Skin Tumors

The research is devoted to the development of a method for identifying skin tumors based on multimodal joint analysis of Raman scattering data and dermatoscopic images. Experimental skin Raman spectra were recorded using a portable setup that includes a laser source with a central wavelength of 785 nm. The spectra were recorded in the range from 792 to 1874 cm–1. Dermatoscopic images of skin neoplasms were obtained using a digital dermatoscope. Machine learning methods, in particular, convolutional neural networks, were used to analyze the registered data. The classification model for malignant melanoma and benign pigmented neoplasms has shown an increase in classification accuracy compared to the analysis of Raman spectra or dermatoscopic images alone. As a result, combined multimodal method for diagnosing skin cancer, which simultaneously takes into account both specific spectral features of neoplasms and spatial inhomogeneities in the distribution of absorbance, has been proposed. The studied approaches to the analysis of optical biopsy data can be further used as part of the software for automated screening diagnostics of skin pathologies in order to detect neoplasms at an early stage of development.