Anomaly Detection in <scp>IoT</scp> Network Traffic Using Bidirectional <scp>3D</scp> Quasi‐Recurrent Neural Network Optimize With Coati Optimization Algorithm

ABSTRACT Anomaly detection in Internet of Things (IOT) network traffic involves identifying abnormal patterns or behaviors, enabling early detection of potential security threats or system malfunctions in the IOT ecosystem. IoT encompasses a variety of networks, addressing not only security challenges within sensor networks, the internet, and mobile communication networks but also specifically focusing on issues related to privacy protection, information management, network authentication, and access control. In this manuscript, Anomaly Detection in IoT Network Traffic using a Bidirectional 3D Quasi‐Recurrent Neural Network with Coati Optimization Algorithm (ADIOT‐B3DQRNN‐COA) is proposed. Initially, the input data are collected from the DS2OS Dataset. Then, the collected data is fed into pre‐processing utilizing an Implicit Unscented Particle Filter (IUPF). The IUPF is used to remove the invalid data. Subsequently, the preprocessed data are sent into the Archimedes optimization algorithm (AOA) to select features. Seven characteristics from the DS2OS dataset are chosen using AOA. The selected features are then fed into a Bidirectional 3D Quasi‐Recurrent Neural Network (Bi‐3DQRNN) to classify anomaly detection in an Internet of Things network into the following categories: data probing, malicious control, malicious operation, scan, spying, incorrect configuration, DOS attack, and normal. To guarantee accurate classification of anomaly detection in IoT networks, Bi‐3DQRNN generally does not express any adaptation of optimization algorithms for figuring out the best parameters. Hence, the Coati Optimization Algorithm (COA) to optimize Bi‐3DQRNN accurately classifies anomaly detection in the IOT network. The proposed ADIOT‐B3DQRNN‐COA approach is implemented in MATLAB. The performance of the proposed method was examined utilizing performance metrics like Accuracy, Computational Time, F‐measure, Precision, Recall, and ROC. The proposed ADIOT‐B3DQRNN‐COA approach contains 32.15%, 28.31%, and 24.18% higher accuracy, 28.69%, 33.54%, and 19.46% higher Precision, and 24.50%, 32.34%, and 24.18% lower Error Rate compared with existing methods, such as Anomaly Detection in IOT network depend on enhanced metaheuristic feature selection optimization and hybrid deep neural network (DNN‐ADIOT‐NTT), Investigation on the stability of DRNN depend on the oversampling method in IDIIOT (DRNN‐PSO‐IDIIOT) and light gradient boosting machine in Anomaly Detection in IOT network: concurrent feature selection with hyperparameter tuning concurrent feature selection with hyperparameter tuning (SMSG‐SCADA‐SNN) respectively.