can-sleuth: Sleuthing out the capabilities, limitations, and performance impacts of automotive intrusion detection datasets

Abstract Modern automobiles are made up of networks of computers, one of which is the inherently insecure Controller Area Network (CAN). Over the years, automotive security has been enhanced by secure gateways and new protocols such as automotive Ethernet, but the CAN protocol has remained the weak link. Automotive researchers have been exploring intrusion detection systems (IDSs) as a potential solution to the problem of CAN bus in security. To build and evaluate an IDS, however, researchers need adequate training and testing data. In this paper, we analyze and compare the following automotive intrusion detection datasets: (1) HCRL Car Hacking, (2) HCRL Survival Analysis, (3) , (4) UNIMORE Bus-Off, (5) UNIMORE DAGA, and (6) UNIMORE Ventus. The two HCRL datasets are well-established in the literature, whereas is a promising new dataset—and the three UNIMORE datasets lie somewhere in between. In our evaluation, we pit sixteen machine learning IDSs against each dataset and analyze the results. In addition, we conduct a feature evaluation of , and we investigate the impact of train-test interdependence in the three UNIMORE datasets. We find that, when pitted against the five comparison datasets, paints a clearer picture of an IDS’s true capabilities; in fact, ’s testing scenarios can reveal when an IDS has overfitted to a particular vehicle type—unlike the UNIMORE datasets. Furthermore, unlike the HCRL datasets, provides more than enough data to train a complex machine learning model—an order of magnitude more—reducing the risk of underfitting. Moreover, maintains ample differentiation power; the standard deviation of the models’ F1-scores was 0.2392 (excluding suppress attacks), whereas the standard deviations for the remaining datasets—HCRL Car Hacking, HCRL Survival Analysis, UNIMORE Bus-Off, UNIMORE DAGA, and UNIMORE Ventus—were 0.2254, 0.2333, 0.1824, 0.2121, and 0.2100 (excluding suppress attacks), respectively.