Designing A Multi-Agent System for Malicious PDF Detection Using Machine Learning and Generative Artificial Intelligence

The growing sophistication of cyber threats exploiting PDF files represents a significant ‎challenge for modern cybersecurity systems. To address this issue, this paper introduces a ‎modular Multi Agent System (MAS) designed for the detection of malicious PDF files. ‎Within this architecture, one agent employs a Conditional Tabular GAN (CTGAN) to expand ‎the training dataset and reduce class imbalance, while another agent integrates supervised ‎machine learning models for classification. Six supervised learning models, namely Decision ‎Tree, Random Forest, XGBoost, Support Vector Machine, Naïve Bayes, and Neural Network, ‎are evaluated on the enriched dataset. Among them, XGBoost achieves the best performance. The MAS coordinates autonomous agents dedicated to dataset management, data ‎augmentation, learning, decision making, and user interaction, ensuring flexibility and ‎scalability under both standard and adversarial conditions. To support interpretability, SHAP ‎analysis is applied during the evaluation phase to identify the features that most strongly ‎influence model decisions. Taken together, the proposed system demonstrates a ‎comprehensive, explainable, and adaptable framework that contributes to strengthening PDF ‎malware detection in sensitive digital infrastructures‎.