Security analysis of the RSSP-II protocol and a novel security enhancement approach

The railway signal safety protocol-Ⅱ (RSSP-Ⅱ), as the core security communication protocol of China’s train control system (CTCS-3), played a crucial role in ensuring the security and reliability of train communication. However, existing research revealed that RSSP-Ⅱ’s message authentication security layer (MASL) had security vulnerabilities like key recovery and forgery, which might endanger the train control system’s stable operation. To address these security issues, the formal modeling method of colored Petri nets (CPN) was employed to analyze RSSP-Ⅱ, and the existence of security vulnerabilities in it was verified. Consequently, an improved protocol, EN - RSSP (enhanced railway signal safety protocol), was proposed. The improvement primarily comprised three aspects. First, a temporary key pair for the communication devices of both parties was dynamically constructed through a trusted third party in a secure channel. Next, session key negotiation was achieved based on the difficulty of the discrete logarithm problem in elliptic curve bilinear pairing combined with a hash function. Finally, a dual-key encryption system made up of the session key and the pre-shared key of the communication devices of both parties was adopted to encrypt and transmit messages. Moreover, timestamps and CMAC (cipher-based message authentication code) were incorporated into the information exchange process to ensure the confidentiality and integrity of messages between devices. Security assessment of EN-RSSP and RSSP-Ⅱ was conducted using the CPN formal analysis tool. The results show that the improved protocol could not only effectively resist key recovery and forgery attacks, but also prevent replay attacks, thereby significantly enhancing the system’s security.