Q-Learning-Based Adaptive Defense Mechanism for Connected Autonomous Vehicles

Connected Autonomous Vehicles (CAVs) are poised to operate alongside human-driven vehicles in mixed autonomy scenarios. However, their reliance on advanced software systems introduces significant cybersecurity vulnerabilities, including code injection and control system exploits, which pose risks to safety and reliability. While reinforcement learning (RL) has been extensively utilized to optimize decision-making in CAVs, its application to cybersecurity remains limited. In this paper, we propose an adaptive defense mechanism that leverages code diversification and allows dynamic switching of the CAV software from one code variant to another, where code variants refer to implementation of CAV software in multiple programming languages. The proposed mechanism is designed based on Q-learning, an RL approach to select the optimal code variant of the software to ensure resiliency. We design the reward function of our Q-learning agent in a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">${way}$</tex> that prioritizes less vulnerable states and penalizes the agent for unnecessary transitions. By incorporating the switching cost in the reward function, we ensure reduced operational cost while enhancing security. This paper demonstrates the potential of RL as a proactive cybersecurity strategy, fostering safer and more resilient autonomous transportation systems.