Multi-Class Segmentation and Classification of Intestinal Organoids: YOLO Stand-Alone vs. Hybrid Machine Learning Pipelines

Background: The automated analysis of intestinal organoids in microscopy images are essential for high-throughput morphological studies, enabling precision and scalability. Traditional manual analysis is time-consuming and subject to observer bias, whereas Machine Learning (ML) approaches have recently demonstrated superior performance. Purpose: This study aims to evaluate YOLO (You Only Look Once) for organoid segmentation and classification, comparing its standalone performance with a hybrid pipeline that integrates DL-based feature extraction and ML classifiers. Methods: The dataset, consisting of 840 light microscopy images and over 23,000 annotated intestinal organoids, was divided into training (756 images) and validation (84 images) sets. Organoids were categorized into four morphological classes: cystic non-budding organoids (Org0), early organoids (Org1), late organoids (Org3), and Spheroids (Sph). YOLO version 10 (YOLOv10) was trained as a segmenter-classifier for the detection and classification of organoids. Performance metrics for YOLOv10 as a standalone model included Average Precision (AP), mean AP at 50% overlap (mAP50), and confusion matrix evaluated on the validation set. In the hybrid pipeline, trained YOLOv10 segmented bounding boxes, and features extracted from these regions using YOLOv10 and ResNet50 were classified with ML algorithms, including Logistic Regression, Naive Bayes, K-Nearest Neighbors (KNN), Random Forest, eXtreme Gradient Boosting (XGBoost), and Multi-Layer Perceptrons (MLP). The performance of these classifiers was assessed using the Receiver Operating Characteristic (ROC) curve and its corresponding Area Under the Curve (AUC), precision, F1 score, and confusion matrix metrics. Principal Component Analysis (PCA) was applied to reduce feature dimensionality while retaining 95% of cumulative variance. To optimize the classification results, an ensemble approach based on AUC-weighted probability fusion was implemented to combine predictions across classifiers. Results: YOLOv10 as a standalone model achieved an overall mAP50 of 0.845, with high AP across all four classes (range 0.797–0.901). In the hybrid pipeline, features extracted with ResNet50 outperformed those extracted with YOLO, with multiple classifiers achieving AUC scores ranging from 0.71 to 0.98 on the validation set. Among all classifiers, Logistic Regression emerged as the best-performing model, achieving the highest AUC scores across multiple classes (range 0.93–0.98). Feature selection using PCA did not improve classification performance. The AUC-weighted ensemble method further enhanced performance, leveraging the strengths of multiple classifiers to optimize prediction, as demonstrated by improved ROC-AUC scores across all organoid classes (range 0.92–0.98). Conclusions: This study demonstrates the effectiveness of YOLOv10 as a standalone model and the robustness of hybrid pipelines combining ResNet50 feature extraction and ML classifiers. Logistic Regression emerged as the best-performing classifier, achieving the highest ROC-AUC across multiple classes. This approach ensures reproducible, automated, and precise morphological analysis, with significant potential for high-throughput organoid studies and live imaging applications.