AI-BASED ALGORITHMS FOR NETWORK SECURITY: TRENDS, PER-FORMANCE, AND CHALLENGES

Abstract: The advancement of network security faces growing challenges as cyberattacks become more sophisticated. Traditional rule-based systems struggle with zero-day attacks and obfuscation techniques. This study examines the development trends of AI-based algo-rithms, particularly machine learning and deep learning, in threat detection. A literature review evaluates AI-driven approaches, including support vector machines, random for-est, deep neural networks, convolutional neural networks, and reinforcement learning. Findings show that AI enhances detection accuracy, adaptability, and reduces false posi-tives. Machine learning efficiently classifies known attacks, while deep learning excels in identifying complex patterns such as distributed denial-of-service and advanced persis-tent threats. Unsupervised learning improves anomaly detection without labeled data. However, AI models require high-quality data, substantial computational resources, and remain vulnerable to adversarial attacks. Despite these challenges, AI provides a dynam-ic and adaptive security solution, surpassing traditional systems. Future research should enhance AI scalability and resilience for evolving cybersecurity threats. Keywords: anomaly detection; artificial intelligence; deep learning; machine learning; network security Abstrak: Perkembangan keamanan jaringan menghadapi tantangan yang semakin besar seiring meningkatnya kompleksitas serangan siber. Sistem berbasis aturan tradisional kesulitan mendeteksi zero-day attack dan teknik penyamaran. Penelitian ini mengkaji tren pengembangan algoritma berbasis AI, khususnya machine learning dan deep learning, dalam deteksi ancaman. Literature review mengevaluasi pendekatan berbasis AI, termasuk support vector machines, random forest, deep neural networks, convolutional neural networks, dan reinforcement learning. Hasil penelitian menunjukkan bahwa AI meningkatkan akurasi deteksi, adaptabilitas terhadap ancaman baru, serta mengurangi false positive. Machine learning efektif mengklasifikasikan serangan yang telah diketahui, sementara deep learning unggul dalam mengenali pola kompleks seperti distributed denial-of-service dan advanced persistent threats. Unsupervised learning meningkatkan deteksi anomali tanpa memerlukan data berlabel. Namun, AI masih bergantung pada data berkualitas tinggi, sumber daya komputasi besar, dan rentan terhadap adversarial attack. Meskipun demikian, AI menawarkan solusi keamanan yang lebih dinamis dan adaptif dibandingkan sistem tradisional. Penelitian selanjutnya perlu difokuskan pada peningkatan skalabilitas dan ketahanan AI dalam menghadapi ancaman siber yang terus berkembang. Kata kunci: deteksi anomali; jaringan keamanan; kecerdasan buatan; pembelajaran dalam; pembelajaran mesin