Comparison of YOLO and transformer based tumor detection in cystoscopy

Abstract Background: Bladder cancer (BCa) is the second most common type of cancer in the genitourinary system and causes approximately 165,000 deaths each year. The diagnosis of BCa is primarily done through cystoscopy, which involves visually examining the bladder using an endoscope. Currently, white light cystoscopy is considered the most reliable method for diagnosis. However, it can be challenging to detect and diagnose flat, small, or poorly textured lesions. The study explores the performance of deep learning systems (YOLOv7- tiny, RT-DETR18), originally designed for detecting adenomas in colonoscopy images, when retrained and tested with cystoscopy images. The deep neural network used in the study was pre-trained on 35,699 colonoscopy images (some from Mannheim) and both architectures achieved a F1 score of 0.91 on publicly available colonoscopy datasets. Results: When the adenoma-detection network was tested with cystoscopy images from two sources (Ulm and Erlangen), F1 scores ranging from 0.58 to 0.81 were achieved. Subsequently, the networks were retrained using 12,066 cystoscopy images from Mannheim, resulting in improved F1 scores ranging from 0.77 to 0.85. Conclusion: It could be shown that transformer based networks perform slightly better than YOLOv7-tiny networks, but both network types are feasable for lesion detection in the human bladder. The retraining of the network with additional cystoscopy data led to an improvement in the performance of urinary lesion detection. This suggests that it is possible to achieve a domain-shift with the inclusion of appropriate additional data.