Enhancing Fibre Bundle Endomicroscopy Through Machine Learning

Endomicroscopy is a medical imaging technique that enables in vivo, in situ imaging of a variety of different areas of the body. There are many approaches to endomicroscopy, we mention several examples: 1) confocal fibre bundle endomicroscopy, often referred to as probe-based confocal microscopy (pCLE), 2) widefield fibre bundle endomicroscopy which uses a fast camera at the distal end (i.e. outside the human body), 3) a method of scanning a single fibre referred to as endoscope-based confocal laser endoscopy (eCLE) as well as 4) chip-on-tip endomicroscopy where a camera is place on the end of an endoscope. The primary focus of this thesis is the fibre bundle endomicroscopy, which works by bundling ~4,000-30,000 fibre cores with core sizes averaging 4μm in size. These cores are surrounded by cladding, which creates a honeycomb pattern of dark areas that limits information and obscures the target. Within endomicroscopy, this may lead to reduced interpretability of the images and potential misdiagnoses in clinical practice, indicating a need to better resolve images obtained in these contexts. Currently, classical reconstruction methods such as Gaussian blurring and linear interpolation are used to fill in these dark areas. This thesis aims to demonstrate a new method using multi-frame machine learning to improve the reconstruction of fibre bundle images. In this thesis a multi-frame machine learning (ML) technique is used to enhance the spatial resolution of synthetic fibre bundle images, by exploiting bundle rotation. The resulting images are evaluated using peak-signal-to-noise-ratio and structural similarity index. The multi-frame ML model improves the quality of the images over standard techniques such as Gaussian blurring and linear interpolation. Multi-frame super resolution in fibre bundle endomicroscopy is not without its drawbacks. An investigation into the limitations of the developed multi-frame ML technique was undertaken. Such as testing the robustness of the model by applying a trained model to a variety of different synthetic fibre bundle core arrangements. The resulting images contained undesired artefacts, the SSIM of the images fell from 0.77 of the original to an average of 0.57 of the four new fibre bundles tested and the PSNR went from 27.77 with the original fibre to 27.46 with the new fibre bundles. Other tests included determining the spatial resolution of the model by reconstructing smaller and smaller sine waves. The multi-frame model could resolve down to 356 cycles/mm compared to that of linear interpolation which could only resolve down to 169 cycles/mm. Finally, testing the model’s ability to reconstruct real fibre bundle images when the model is trained only using synthetic fibre bundle images. These experiments demonstrated that the model presented was not robust enough to overcome the limitations but gave valuable insight to chart the direction of future research. Finally, EndoscoPY, an ML-based open-source software was created that other researchers can use and build upon for their specific needs. The software was tested using two different ML models and using both real and synthetic fibre bundle imaging techniques. The scalability of the software, as well as the machine learning model was tested using two larger datasets. It was found that as the resolution of the input image dataset is increased the time taken for the machine learning model to be applied also increased. The 512×512 images took 106ms on average for MFAE and the 768×768 taking 193ms on average. While 512×512 results in an average of 10fps, still usable in practice, the larger 768×768 would be unfit for real-time application. The work done in this thesis demonstrates that multi-frame ML reconstruction can provide an image of higher quality than that of classical methods such as Gaussian blurring and linear interpolation. The reconstructed images are also of a higher accuracy, as proven by measuring the cell nuclei size of reconstructed tissue images. Finally, a custom-made software is presented as capable of performing real time fibre bundle reconstruction using different ML models, programmed using open-source programming libraries to allow for wider usability and adaptation of this method.