Toward Dynamic Data-Driven Time-Slicing LSH for Joinable Table Discovery

In legacy industrial systems, discovering joinable information between database tables is important for applications such as data integration and data analysis. Locality-Sensitive Hashing-based methods have been proven to be capable of handling chaotic and diverse table relationships, but these methods often rely on an incorrect assumption—that the similarity of table columns in the database directly reflects their joinability, causing problems related to the accuracy of their results. To solve this problem, this study proposes a dynamic data-driven time-slicing Locality-Sensitive Hashing method for joinable table discovery. This method introduces database log information and within different time slices, uses the co-occurrence matrix of data tables to determine their joinability. Specifically, it first performs a MinHash dimensionality reduction on database columns and then uses Locality-Sensitive Hashing to calculate the static similarity. Next, it identifies business modular time slices through database logs, calculates the dynamic similarity of the slice time data, and builds a co-occurrence matrix between tables. Finally, the joinability between data tables is calculated using the static similarity, dynamic similarity, and co-occurrence matrix. The experimental results demonstrate that this method effectively excludes tables that only have similarity but no business relationship for data cleaning, and its accuracy exceeds that of methods that only depend on similarity.