ACOD: An Asynchronous Clustering Framework Based on Outlier Detection for Data Lakehouse

With the development of big data technology, the amount of data available to us is growing by leaps and bounds, and Data Lakehouse technology has become a mainstream solution. It is important to note that Data Lakehouse provides an important data clustering mechanism, but this also brings problems of cluster resource waste, unsupported concurrent updates, and unfriendly incremental queries. To address these issues, we propose an asynchronous clustering framework based on outlier detection. It uses improved outlier detection algorithms and metadata information to analyze and determine tasks that may be anomalous and then optimizes the data layout through the data clustering mechanism. From the experimental results, the improved outlier detection algorithm performs better, which can control the number of times the clustering operation is executed and mitigate the problems of unsupported concurrent updates, unfriendly incremental queries, and wasted cluster resources brought by the clustering mechanism.