Advances in NILM dataset evaluation: a comparative review within the NILMTK framework

Abstract This study presents a comparative and chronological review of energy datasets fundamental to research in Non-Intrusive Load Monitoring (NILM). Aiming to assist researchers in selecting appropriate datasets aligned with their objectives, the article systematically benchmarks prominent public datasets, including recent ones such as DEPS, DSUALM, DSUALM10H, and OMPM/UALM2, using the open-source NILM toolkit (NILMTK) to facilitate reproducibility and comparison results. The chronological evaluation highlights the evolution of dataset features, from early limitations in standardization to the recent inclusion of high-frequency sampling and harmonic information enabled by open hardware platforms. Widely used NILM algorithms are evaluated across datasets employing various NILMTK metrics, including F1-score, MAE, RMSE, and EAE, revealing an average F1-score of 0.6135. Traditional approaches, such as the Factorial Hidden Markov Model, achieve the highest average F1-score (0.6600), demonstrating robust efficiency. Conversely, deep learning models, such as Recurrent Neural Networks, show superior accuracy with lower MAE (111.26 W) and RMSE (180.98 W) on specific datasets, but at a significantly higher computational cost. Results indicate significant variability in algorithm performance depending on dataset specifics and device types. Appliances with stable consumption patterns, such as refrigerators and LED lamps, demonstrate higher accuracy (e.g., F1-score of up to 0.72 for refrigerators in Dataport and 0.78 for LED lamps in UALM2). In contrast, complex or multistate devices, such as washing machines and electric heaters, present greater challenges (e.g., an F1-score of 0.0 for washing machines in iAWE and a high MNEAP for electric heaters in REFIT). Overall, NILM dataset selection depends not on a universally superior resource, but on the alignment of dataset attributes (sampling frequency and duration, number and variety of monitored homes, types of measurements, including harmonics and contextual data) and compatibility with NILMTK, tailored to the research objectives and problem complexity. While traditional algorithms offer computational efficiency, deep learning models provide greater accuracy at a higher cost. The use of open hardware platforms such as OpenZmeter and OMPM promotes reproducibility and accelerates NILM innovation. This review emphasizes the importance of striking a balance between study goals, technical resources, and measurement tools to establish a robust framework for algorithm development and validation in energy disaggregation.