Enhancing One-Class Anomaly Detection in Unlabeled Datasets Through Unsupervised Data Refinement

Effective anomaly detection in unlabeled datasets is crucial for numerous applications across different domains. This paper presents a novel methodology for one-class anomaly detection that leverages unsupervised learning techniques to filter and purify normal data which leads to enhancing the training process in real world scenarios where no labels are available. The initial phase employs an Isolation Forest (IF) algorithm to segregate preliminary normal data points from potential outliers. To refine the selection of truly normal data, we introduce a centroid-based filtering method where distances from each point to the data centroid are calculated and analyzed. We then select a subset of these points, ensuring their purity as representative normal data points. This purified dataset forms the basis for training a one-class Support Vector Machine (SVM), Isolation Forest (IF) and Auto-Encoder (AE), focusing solely on these selected normal points. Our methodology was tested across two distinct public real-world datasets, evaluating performance metrics such as F1-Score, Precision and Recall validating the effectiveness of our approach. Results indicate that the refined training data set contributes to improved models' performance up to 20%. This research contributes to the field of anomaly detection by providing a systematic approach to handling and training on unlabeled data, thereby proposing a scalable solution for real-world anomaly detection challenges.