Explainable Deep Hybrid Learning for Early Prediction and Multi-Class Classification of Knee Osteoarthritis Using X-Ray Images

Objectives: To propose a deep hybrid model using the ResNet34 and Vision Transformer to classify the severity of KOA using X-ray images. Method: Knee Osteoarthritis (KOA), being one of the most common and progressive conditions in the field of orthopedics, is posing a significant financial challenge to the healthcare sector globally. The study employed the publicly available Osteoarthritis Initiative (OAI) dataset, comprising 8,260 knee X-ray images partitioned into pre-defined training (n=5,778), validation (n=826), and test (n=1,656) splits. CLAHE preprocessing (clip limit 2.0; tile grid 8×8) was applied to all images prior to training. A hybrid ResNet34 and Vision Transformer architecture was trained end-to-end on five-class KL grading under a class-weighted cross-entropy objective, with AdamW optimisation and cosine annealing scheduling. Findings: The proposed model, ResNet34-ViT achieved the highest performance with an accuracy of 66.79%. Compared to other models, DenseNet-121 and ResNet-50 V2 reached 66.31% and 65.36% accuracy, respectively. Class-balanced oversampling (10,000 training samples per KL grade) and inverse-frequency loss weighting improved minority-grade recall. Grad-CAM++ heatmaps aligned model attention with joint-space narrowing, osteophyte formation, and subchondral changes across KL grades. However, the model still has limitations, indicating the need for further research to enhance its clinical applicability. Novelty: This study introduces a hybrid model combining ResNet34 and Vision Transformer for classifying knee osteoarthritis. A targeted preprocessing pipeline, anchored by Contrast-Limited Adaptive Histogram Equalization (CLAHE; clip limit 2.0, tile grid 8×8) and class-specific oversampling, resolves a 13.21× distributional imbalance present in the OAI dataset, enabling equitable five-category grading across all Kellgren-Lawrence severity tiers. Grad-CAM++ heatmaps further validate model decisions by aligning activation regions with anatomical structures that radiologists routinely examine during KL grading, including the joint space, subchondral bone, and osteophyte margins. Keywords: Knee Osteoarthritis (KOA), KL Grading, X-Ray Images, Convolutional Neural Network (CNN), Vision Transformer (ViT), GradCAM++