Real-time marker-less needle tracking for CT-guided interventions using multiple RGB cameras

PURPOSE: In CT-guided interventions, instrument tracking reduces patient risk while improving the overall outcome. Though useful, marker-based tracking systems suffer from occlusion, are complex to set up, and fail when needles deflect. A marker-less tracking system could overcome these challenges, enabling streamlined integration. METHODS: We developed and evaluated a proof-of-concept system combining standard RGB cameras with a U-Net-ConvNeXt architecture for needle detection and tracking. We compiled a fully annotated dataset of 35,000 frames, spanning multiple needle types and scenarios. We evaluated tracking accuracy against state-of-the-art marker-based systems, utilizing an IR-based and an image-based system on static positioning, dynamic movement, and needle-deflection scenarios. Moreover, our marker-less system was evaluated using both known and unknown needle types. RESULTS: mm). CONCLUSION: Our proof-of-concept demonstrates the feasibility of marker-less needle tracking, suggesting potential for clinically relevant accuracy, reliability, and reduced complexity while exhibiting robustness to needle deflection and varying needle appearances. These preliminary results indicate promising directions for future clinical development.