An Efficient Mutual Authentication and Fractional Lyrebird Optimization With Deep Learning–Based SIP‐Based DRDoS Attack Detection

Over the past decades, Session Initiation Protocol (SIP) has been commonly utilized in signaling protocol to control and signal multimedia communication sessions for providing various services to users. Video calls and internet telephony over Internet Protocol (IP) are the most common applications on SIP. Generally, the SIP acts as an unsafe communication platform, due to the exchange of information via public channels. Thus, the security of the SIP network should be increased prominently and require secure authentication and key establishment schemes. In this article, a deep learning model fractional lyrebird optimization algorithm–deep stack autoencoder (FLOA‐DSA) is developed for the detection of SIP‐based distributed reflection denial‐of‐service (DRDoS) attacks in SIP network. Here, the authentication process is carried out under different phases based on various mathematical model functionalities. Moreover, the deep stack autoencoder (DSA) model is used to perform intrusion detection in the attack validation phase. The detection performance of DSA is increased by training using the fractional lyrebird optimization algorithm (FLOA); FLOA provides a more effective, reliable, and scalable optimization strategy for training DSAs than traditional algorithms. Its ability to dynamically interact with DSA’s learning dynamics makes it not just a suitable choice, but the optimal optimization choice for this hybrid deep learning system. Further, the detection performance of the FLOA‐DSA is evaluated based on the metrics, such as detection rate, F ‐measure, false positive rate (FPR), memory usage, validation time, conditional privacy, response time, energy efficiency, and latency, and the FLOA‐DSA obtained best results of 90.12%, 91.40%, 9.13%, 6.9 MB, 48.5 s, 3.86 s, 88.88 bits/J, and 2.68 s. The F ‐measure of the proposed method is 9.06%, 6.63%, 4.80%, and 2.61% improved than other existing comparative methods.