SOE: A Multi-Objective Traffic Scheduling Engine for DDoS Mitigation with Isolation-Aware Optimization

Distributed Denial-of-Service (DDoS) attacks generate deceptive, high-volume traffic that bypasses conventional detection mechanisms. When interception fails, effectively allocating mixed benign and malicious traffic under resource constraints becomes a critical challenge. To address this, we propose SchedOpt Engine (SOE), a scheduling framework formulated as a discrete multi-objective optimization problem. The goal is to optimize four conflicting objectives: a benign traffic acceptance rate (BTAR), malicious traffic interception rate (MTIR), server load balancing, and malicious traffic isolation. These objectives are combined into a composite scalarized loss function with soft constraints, prioritizing a BTAR while maintaining flexibility. To solve this problem, we introduce MOFATA, a multi-objective extension of the Fata Morgana Algorithm (FATA) within a Pareto-based evolutionary framework. An ϵ-dominance mechanism is incorporated to improve solution granularity and diversity. Simulations under varying attack intensities and resource constraints validate the effectiveness of SOE. Results show that SOE consistently achieves a high BTAR and MTIR while balancing server loads. Under extreme attacks, SOE isolates malicious traffic to a subset of servers, preserving capacity for benign services. SOE also demonstrates strong adaptability in fluctuating attack environments, providing a practical solution for DDoS mitigation.