COMBINING SEGMENTATION, TRACKING, AND CLASSIFICATION MODELS TO SOLVE VIDEO ANALYTICS PROBLEMS

The task of detecting obstacles in front of a mobile robot has been successfully solved long ago usinglaser and ultrasonic sensors. However, obstacles that are not detected by these types of sensors may endangerthe safety of the robot. To detect them in the work, it is proposed to use a technical vision system(STZ), the information from which is processed by a semantic segmentation neural network, which returnsthe mask of the obstacle on the frame and its class. The basis for such a network was the SAM universal segmentationnetwork, which requires further development to be applied to the semantic segmentation task.The peculiarity of this network is its universal applicability, that is, the ability to select any objects in anyfilming situation. At the same time, SAM does not predict the semantics of the object. In this paper, an additionalmodule is proposed that makes it possible to implement semantic segmentation by classifying the featuresof the selected objects. The possibility of using such a module to solve the problem of supplementing thenetwork output with new information is substantiated. The classification result is then fed into the same filteringalgorithm as the masks to ensure consistency between the result of the universal network and the complementarymodule. After integrating the module with the model, a new semantic segmentation model wasobtained, called RTC-SAM in the work. It was used to perform semantic segmentation of a publicly availabledataset with images of an open area. The 45% result obtained by the IoU metric exceeds the result of existingmethods by 13%. The images of the results of using the new network shown in the work make it possible toverify its performance. It also describes the testing of the developed solution with a study of the performanceof the developed model on a PC and a mobile computer. The algorithm on the mobile computer shows insufficientspeed to enter real-time mode – more than 3.5 seconds to process one frame. In this regard, one ofthe directions of further research in the field of improving system performance.