A hybrid machine learning intrusion detection method for metamorphic malware

Abstract Malicious software conduct using sophisticated techniques such as obfuscation and anti-analysis methods to avoid discovery is becoming more complex for digital forensic analysts to deal with. Using these advanced techniques, it has previously been reported that attackers have successfully bypassed existing security measures, making it a constant race to update and improve existent Intrusion Detection Systems (IDSs) and saving billions of dollars in currency of losses. Traditional Signature-based IDSs (SIDSs) and Anomaly-based IDSs (ABIDSs) struggle to detect new and unknown malicious threats, which require more advanced IDSs. These challenges highlight the need for continuous research and development in IDSs to keep up with the evolving security threat landscape. This paper proposes a two-stage solution Machine Learning (ML) detection-method approach. The novelty also arises from the combination of two distinct sets of features that enhance the final outcome more effectively than if they were applied separately. In stage 1, using sequences of opcodes; a discrete Hidden Markov Model (dHMM) validates the input data set; while stage 2 uses the Portable Executable (PE) sections from executable files as features of a Random Forest (RF) . Ultimately, the RF is responsible for the classification and detection purposes of metamorphic malware. This hybrid approach provided promising results with a precision rate of 100% and accuracy of 95%.