Machine Learning Approaches for Detecting Encrypted and Compressed Malicious Files: A Review

The advancement of cyber threats has intensified with the rise of encrypted and compressed malicious files, posing significant challenges to traditional signature-based detection methods. Machine learning (ML) has emerged as a promising solution, capable of uncovering hidden patterns without relying on known malware signatures. This review explores recent developments in applying ML techniques such as deep learning, transfer learning, ensemble methods, and anomaly detection to identify encrypted malicious files. Studies have demonstrated that ML models can effectively analyse structural, statistical, and behavioural features, offering improved adaptability and detection accuracy. Despite these advancements, significant challenges persist, including encryption evasion, polymorphic malware, data imbalance, scalability issues, and adversarial attacks. Innovative strategies like explainable AI, federated learning, scalable architectures, and continuous learning are being investigated to enhance robustness and transparency. The dynamic evolution of cyber threats underscores the urgency of developing intelligent, adaptable, and privacy-conscious detection systems. Ultimately, this review highlights the critical role of ML in strengthening cybersecurity defences against increasingly sophisticated encrypted malicious files, emphasizing the need for continued research to address ongoing challenges and ensure resilient, scalable protection across diverse digital environments.