HyperEye: A Lightweight Features Fusion Model for Unknown Encrypted Malware Traffic Detection

Recently, effectively identifying encrypted malicious traffic without decryption in consumer applications relies heavily on high-quality labeled traffic datasets. However, this harms models for incorrect labeling and requires more efficient real-time identification of encrypted unknown ones. This paper proposes HyperEye, a real-time, unsupervised, encrypted malicious traffic detection system. It can detect unknown traffic patterns by analyzing the fused traffic features in-depth. Precisely, we extract protocol-agnostic numerical and protocol-specific text features and devise a cross-term fusion algorithm to obtain a comprehensive traffic behavior description. We designed a genetic algorithm-based DBSCAN (GA-DBSCAN) parameter optimization algorithm to enhance the quality and stability in identifying malicious traffic. Extensive experimental results with open-world and real-world datasets demonstrate that our work outperforms other state-of-the-art malware detection systems, achieving an average 11.95% improvement in the F1-score. Besides, experimental results with the real-world dataset demonstrate that our system applies to the dynamic nature of consumer applications and can safeguard users’ data and privacy.