A Multi-Model Automated Framework for Scalable and Robust Outlier Detection in Data Warehousing

Data quality is an important aspect that must be considered to obtain reliable and useful data from data warehouses. This paper presents an automated system using ensemble-based outlier detection methods to detect outliers in large data warehouses. The proposed framework combines several unsupervised machine learning algorithms such as Local Outlier Factor (LOF), Isolation Forest (iForest), One-Class SVM (OCSVM), and Autoencoder to improve the effectiveness of the outlier detection process. Using all these models ensures that the framework solves the drawbacks of single methods in scalability and suitability for large and high-dimensional dynamic data. In addition to functioning as an outlier detection method, the framework employs SHAP (Shapley Additive explanations) to identify the causes of the outliers to provide a more in-depth analysis of such outliers. This capability enables organizations to see patterns, possible areas of inefficiency that lead to an outlier, and areas of data inconsistency that need to be corrected. The effectiveness of the system has been verified using the AdventureWorksDW2019 data warehouse, showcasing its resilience in detecting intricate irregularities and tracking their sources across many fact tables. This study provides useful recommendations for enhancing data quality in a range of data warehousing applications and lays the groundwork for scalable outlier identification.