ON THE SIMILARITY FUNCTION OF GRAPHIC REPRESENTATIONS OF EXECUTIVE FILES IN THE OBFUSCING TRANSFORMATION EVALUATION MODEL

Obfuscation of program code is used to complicate its analysis in a model when the analyst has full access to the program. Obfuscation is usually divided into cryptographically secure and heuristically resistant. In the first case, the complexity of the analysis is comparable to the difficulty of solving some known mathematical problem. In the second case, the resistance is usually justified by the lack of effective techniques for analyzing the obfuscation method known at the time of its creation. Cryptographically secure obfuscation has not yet found practical application, while heuristically resistant is widely used. Previously, the authors proposed a model for assessing the efficiency and resistance of heuristic obfuscating transformations based on the use of a similarity function. In this paper, such a similarity function is constructed using machine learning methods based on a comparison of the graphical representation of program executable files. In particular, the comparison is performed using a convolutional network with four convolutional layers, an RMSprop optimizer, an NLLLoss loss function, and two outputs of a fully connected layer. The proposed function is used in the implementation of a model for evaluating the efficiency and resistance of obfuscating transformations. In addition to the similarity function, the implementation of the model also includes: a basic set of obfuscating transformations provided by the Hikari obfuscator; a set of obfuscating transformation sequences based on the basic set; a test set of programs for training models based on the CoreUtils, PolyBench and HashCat program sets; approximation of the most "understandable" version of the program using the smallest version of the program (searched among the versions obtained using various optimization options of the GCC, Clang and AOCC compilers); a program deobfuscation scheme based on the optimizing compiler from LLVM. The results of an experimental study with the implemented model showed that it is impractical to use the constructed similarity function in the framework of the evaluation model due to its low accuracy, but it is possible to use it when constructing more complex functions.