On the Study of One Way to Detect Anomalous Program Execution

Developing more accurate and adaptive methods for detecting malicious code is a critical challenge in the context of constantly evolving cybersecurity threats. This requires constant attention to new vulnerabilities and attack methods, as well as the search for innovative approaches to detecting and preventing cyber threats. This paper examines an algorithm for detecting the execution of malicious code in the process of a protected program. This algorithm is based on a previously proposed approach, when the legitimate execution of a protected program is described by a profile of differences in the return addresses of the called functions, also called a distance profile. A concept is introduced called positional distance, which is determined by the difference between the call numbers in the program trace. The main change was the ability to add to the profile the distances between the return addresses of not only neighboring functions but also several previous ones with the given positional distance. In addition to modifying the detection algorithm, this study develops a tool for automating the construction of a distance profile and experimentally studies the dependence of the probability of false detection of an atypical distance on the training duration for four well-known browsers. The experiments confirm that with a slight increase in verification time, the number of atypical distances detected by the proposed algorithm can be significantly lower than the number of atypical distances detected by the basic algorithm. However, it should be noted that the effect of the transition from the basic algorithm to the proposed one, as the results show, depends on the characteristics of the specific program being protected. The study highlights the importance of continually improving malware detection techniques to adapt them to changing threats and software operating conditions. As a result, this will ensure more reliable protection of information and systems from cyber attacks and other cyber threats.