VIT-DETR: A Hybrid Vision Transformer and Detection Transformer for Hand Fracture Detection and Classification

The challenge of finding the accurate fracture point and the type of fracture has been a challenging task till today, in an emergence situation the accurate detection or classification of fractures in x-ray image are crucial for effective diagnosis and treatment planning. In this paper we introduce an hybrid model that will have an integration of vision transformer and detection transformer to overcome the challenges faced over traditional methods of fracture classification and detection. Bone fractures are commonly diagnosed using X-ray images, but accurate fracture detection and classification remain challenging, especially in emergency situations. Traditional methods often depend on radiologists' expertise, which can lead to inconsistent diagnoses. In this paper, we propose a hybrid model that integrates Vision Transformers (ViT) and Detection Transformers (DeT) to enhance the accuracy of fracture detection and classification. ViT excels at capturing global image features, while DeT focuses on precisely localizing fracture points within the X-ray images. The hybrid approach leverages the strengths of both models, resulting in superior performance in detecting and classifying various fracture types, including subtle and complex fractures. The model is trained on a large dataset of bone fracture X-ray images, achieving high accuracy in both fracture detection and localization. Our approach significantly outperforms conventional CNN-based methods, especially in complex cases where precise localization is crucial. Additionally, the proposed system can assist radiologists in emergency settings, reducing diagnostic time and errors. The hybrid model offers robust and efficient fracture detection, which has the potential to improve clinical decision-making and patient outcomes. Future work will focus on real-time implementation and exploring the model’s applicability to other medical imaging tasks.