Tri-channel visualised malicious code classification based on improved ResNet

As malicious code attacks continue to evolve, attackers leverage techniques like packing and code obfuscation to generate numerous variants, challenging traditional detection methods. Addressing the limitations of current deep learning-based malicious code classification approaches in feature extraction and accuracy, this paper introduces an innovative RGB visualization detection method based on a hybrid multi-head attention mechanism. Initially, a feature representation method utilizing RGB images is introduced. This approach focuses on semantic relationships between a malware’s binary information, assembly details, and API data, generating images with richer textural information. This technique effectively uncovers the deep dependencies between the original and variant versions of malicious code, providing stronger support for subsequent classification tasks. Furthermore, to tackle the issues of malware encryption and obfuscation, a deep neural network framework is adopted, incorporating a modular design philosophy and integrating a multi-head attention mechanism. This design not only enhances the expressiveness of critical features but also helps the model better focus on key aspects of the malicious code, thereby improving classification accuracy. Through comparative experiments and in-depth analysis, the effectiveness and superiority of the proposed RGB visualization method and MSA-ResNet model in the field of malicious code variant classification are validated. The accuracy rates achieved on the Kaggle and DataCon datasets are 99.49% and 97.70%, respectively, representing significant improvements over other methods. This approach demonstrates strong generalization capabilities and resistance to obfuscation, offering a new and effective tool for malicious code detection.