Towards Robust Malware Family Labeling: A Multi-Engine-Tag Discrepancy-Aware Framework

Malware detection and classification predominantly rely on machine learning models, which necessitate the provision of authoritative family labels in the annotated data used for model training. Within the scope of extant research methodologies, the formulation of such authoritative family labels is customarily derived from virus labels furnished by a plurality of antivirus engines, procured through a crowdsourcing methodology. A pivotal challenge that necessitates resolution in this context pertains to the incongruity in virus nomenclature across diverse engines. Prevailing strategies frequently resort to a majority voting mechanism to mitigate this discrepancy, yet they fail to account for the interdependencies and reference relationships that exist among the engines. This manuscript introduces a Multi-Engine-Tag Discrepancy-Aware Framework (MET-DAF), designed to facilitate the construction of malware family labels while addressing the inherent biases present in multi-engine labeling systems. The framework initially delineates the reference relationships between antivirus engines by analyzing label co-occurrence patterns. Following this, engines implicated in reference relationships are systematically excluded, and a crowdsourcing strategy is employed to establish authoritative family labels. The robustness and efficacy of our proposed framework have been rigorously assessed and corroborated across a triad of datasets: the publicly accessible Malheur and MalGenome datasets, as well as a proprietary dataset, CollectData, comprising authentic network traffic data amassed by our research collective.