How Far Does Entropy Go? Duplicates, Confounding, Drift and Padding in Entropy-Based Detection of Malicious PE Files

Abstract Shannon entropy is a common static indicator of packed or encrypted Windows executables. We measure how much the entropy of the first PE section, alone or with the size fields of the same header, tells about maliciousness, on 43,139 PE section headers and 799,920 labelled EMBER 2018 samples. In both datasets 64% of malicious files share their first section with other files, so random cross-validation largely measures memorisation: grouping identical sections lowers the AUC of an entropy-only model from 0.87 to 0.76 and from 0.77 to 0.70. On the EMBER temporal split, 97.5% of the detections of this model at 1% false positives are sections already seen in training; on unseen sections it detects 0.9% of malware, and 7.9% with the size fields added. A strong size confound in the first dataset does not replicate on EMBER, and empty-on-disk first sections are also frequent in benign software. Using standard bounds on the entropy of a section after appending arbitrary bytes, we compute from header data alone which detections are evaded whatever the section content. On the deduplicated dataset, appends guarantee evasion for 52--92% of the samples detected by an entropy-and-size model, and for 96--98% if the attacker can also use padding of benign-looking entropy; appends that stay within the alignment slack or keep the header consistent achieve only 6--24%. Since information content, size times entropy, never decreases under appends, a monotone model is provably immune, but its AUC falls to 0.52--0.62. Adversarial training reduces evasion but does not restore detection.