Action-Infused Reinforcement Learning for Predicting Vascular Ultrasound Images

Ultrasound, as a safe, cost-effective, real-time imaging diagnostic tool, has been widely utilized in medical examinations. In recent years, research has focused on utilizing reinforcement learning (RL) and 3D vascular reconstruction methods to achieve robot-assisted ultrasound scanning. In robot-assisted scanning, predicting the ultrasound images and features corresponding to the robot’s next action helps the agent make better action decisions and achieve scanning goals more efficiently. To this end, we propose a reinforcement learning (RL) framework using the Advantage Actor-Critic (A2C) algorithm to predict ultrasound images and incorporate a LSTM module to leverage temporal information from adjacent time points. To validate the algorithm’s effectiveness, we constructed virtual and real environments to collect scanning data for agent training. In ultrasound vascular scanning, the focus is often on the relationship between the vessel’s position and shape in ultrasound images as the probe’s position changes. To extract relevant information from ultrasound images, we employ an ellipse fitting method for feature extraction and train a Unet network in a real environment for vessel segmentation in ultrasound images. By collecting vascular ultrasound scanning data and inputting it into the RL agent network for training, we can predict the ultrasound image information corresponding to the probe’s position in the next time point, given the probe’s position and ultrasound images from the previous N time points.