A Real-Time Streaming Hybrid Ensemble SOAR Framework Using Kafka Streaming for Autonomous Cyber Anomaly Detection

Modern networks generate large volumes of security logs, making it difficult for analysts to detect and respond to threats in time. In this work, we design and evaluate a real-time streaming SOAR framework that combines a Kafka-based ingestion pipeline with a hybrid ensemble consisting of XG Boost, Random Forest, and an Isolation Forest model. The system processes events as they arrive, assigns risk levels through a weighted consensus score, and triggers automated responses when high- severity activity is detected. Although our evaluation uses simulated enterprise traffic, the architecture reflects real deployment constraints and is implemented in a containerized setup to measure latency and through- put. Experimental results show that the ensemble achieves high detection accuracy while maintaining sub-second processing latency under moder- ate load. We also include ablation studies to understand the contribution of each model. The findings highlight the potential of lightweight ensemble techniques and streaming pipelines for building practical, automated cyber defense systems.