Evaluating and enhancing intrusion detection systems in IoMT: The importance of domain-specific datasets

The emergence of the Internet of Medical Things (IoMT) is revolutionizing healthcare delivery, but also introducing critical challenges to cybersecurity and patient safety. Intrusion Detection Systems (IDSs) enhanced by Machine Learning (ML) have emerged as a powerful solution to identify cyberattacks in these environments. However, existing studies often rely on general IoT datasets, potentially limiting their applicability in IoMT-specific scenarios. This study addresses these limitations by comparing the performance of ML models trained on a general IoT dataset (CICIoT2023) and an IoMT-specific dataset (CICIoMT2024) to demonstrate the importance of domain-specific data. Our findings reveal substantial drops of up to 66.87% in the F1-score when models trained on one dataset are tested on the other. Furthermore, the study critiques key dataset design choices in CICIoMT2024, and proposes baseline optimization techniques including uniform windowing, proper train-validation-test splits, adjustments in temporal dependencies for time series data, and improved dataset balancing. By applying these techniques, we observe significant improvements in IDS performance in comparison to other approaches, with scores of 0.9985 in model accuracy. The findings show the necessity of using IoMT-specific datasets and carefully designed preprocessing techniques to build robust IDSs tailored to the unique demands of medical IoT environments.