Optimized hybrid BEiT and EfficientNetB0 deep learning model with whale and particle swarm algorithms for colorectal cancer histopathological image classification

Abstract One of the deadly cancers, known as colorectal cancer, affects millions of people globally and therefore the need for an effective way of diagnosing it at early stages. While histopathology is the current method of diagnosis of colorectal cancer, it is time-consuming and subject to human interpretation whose results will vary according to the individual carrying out the diagnosis. Although deep learning has been successful in automating the process, traditional CNNs cannot extract global context information, and Vision Transformers must be optimized in order to obtain consistent results. To overcome such limitations, this study will develop a deep learning hybrid model using the strengths of both the Bidirectional Encoder Representations from Image Transformers and EfficientNetB0 in classifying colorectal cancer. This process will be optimized by Whale Optimization Algorithm (WOA) and Particle Swarm Optimization (PSO). For this reason, the experiment will use 5000 hematoxylin and eosin-stained images of eight different types of tissues to train the models. From the results obtained, the optimized hybrid model performed better compared to the baseline hybrid model with 96.00% and 96.27% accuracy for WOA and PSO, respectively, against 92.13% for the baseline hybrid model. Among the two optimization methods, PSO outperformed WOA. In this connection, the integration of the Transformer model’s feature learning, convolution feature extraction, and bio-inspired algorithms proves to be a successful approach for automated diagnosis of colorectal cancer. The methodology can be helpful in aiding the pathologists with uniform diagnosis and decision-making process.