Smart surgical telementoring system with patch-to-pixel point tracking method

The utilization of surgical telementoring has become increasingly prevalent in enhancing the surgical standards of grassroots hospitals in contemporary times. In the traditional framework of surgical telementoring, remote doctors guide grassroots doctors via video and audio communication, which may encounter obstacles such as inadequate information transfer and challenges in accurately pinpointing the surgical site and path. To mitigate these issues, this study introduces an intelligent surgical telementoring system based on edge computing. This system enables the transmission of points marked by remote doctors to grassroots doctors, updating these points’ coordinates in real-time on the local endoscopic video. In this system, a novel method named P2PTracking (Patch-to-Pixel Point Tracking) is implemented. This process begins with the tracking of a square patch surrounding the point marked by the remote doctor using SiameseFC. Following this, feature matching is performed on the tracked patch in the current frame and the square patch in the labelled frame or template frame. The affine transformation is then calculated based on the feature matching results. Lastly, the point tracking result is derived using the computed affine transformation. Experimental results indicate that the proposed system has a transmission speed of 1.99M/s and a transmission latency of 171 ​ms when transmitting video at a resolution of 1920 ​× ​1080px, while the proposed method can achieve an accuracy of 96.6 ​% when the pixel error is 4. The code and data are available at https://github.com/hfut66/P2PTracking.git .