The Effect of Generating Synthetic Data in Smart City Network Systems

This study examines the effect of synthetic data generation for balancing class distributions on the performance of classification algorithms in smart city network systems. Contrary to the assumption that data balancing improves classification performance, the analysis reveals a more complex impact. Using three publicly available network traffic benchmark datasets and four different balancing techniques, the study evaluates the performance of five classifiers on 65 classification tasks. The findings indicate that, for smaller datasets, classifiers that achieved the highest accuracy on unbalanced data did not benefit from synthetic data generation for minority classes. Although neural network-based classifiers showed improved performance with balanced data, these improvements came at the cost of lower overall classification scores. For larger datasets, balancing through random oversampling of minority classes and undersampling of majority classes helped improve classification. However, these improvements were limited to precision, with no significant gains in recall. The study offers valuable insights into using synthetic data for intrusion detection, emphasizing the challenges of intricate dependencies in network traffic data for generative models. The results align with previous research showing mixed effects of data balancing on classifier performance, contributing to a broader understanding of the limited efficacy of synthetic data in real-world network contexts. This experimental study highlights the need for a systematic benchmarking framework for synthetic data research, ensuring consistency in data balancing and classification processes. This work contributes to the ongoing discourse on the intersection of machine learning and cybersecurity, emphasizing the critical role of data in developing resilient intrusion detection systems.