GRM-083 Leveraging Graph Attention Networks and BERT for Robotic Surgery Report Generation

This project focuses on generating surgical reports from robotic surgery videos by leveraging graph-based representations of instrument-tissue interactions. We utilize Graph Attention Networks (GAT) to model these interactions, which are then integrated into a BERT-based language model for caption generation. Our approach enhances the accuracy of automated surgical reporting by capturing spatial and relational dependencies within surgical scenes. The model is evaluated on the Robotic Instrument Segmentation dataset from the 2018 MICCAI Endoscopic Vision Challenge(Endovis-18) and TORS surgery dataset, achieving high performance across multiple metrics, including BLEU-n, Cider, and ROUGE scores. By automating report generation, this study aims to assist healthcare professionals in improving post-surgical care, optimizing procedural efficiency, and enhancing decision-making in robotic-assisted surgeries.