Enhancing data curation with spectral clustering and Shannon entropy: An unsupervised approach within the data washing machine

In the realm of digital data proliferation, effective data curation is pivotal for extracting meaningful insights. This study explores the integration of spectral clustering and Shannon Entropy within the Data Washing Machine (DWM), a sophisticated tool designed for unsupervised data curation. Spectral clustering, known for its ability to handle complex and non-linearly separable data, is investigated as an alternative clustering method to enhance the DWM’s capabilities. Shannon Entropy is employed as a metric to evaluate and refine the quality of clusters, providing a measure of information content and homogeneity. The research involves rigorous testing of the DWM prototype on diverse datasets, assessing the performance of spectral clustering in conjunction with Shannon Entropy. Results indicate that spectral clustering, when combined with entropy-based evaluation, significantly improves clustering outcomes, particularly in datasets exhibiting varied density and complexity. This study highlights the synergistic role of spectral clustering and Shannon Entropy in advancing unsupervised data curation, offering a more nuanced approach to handling diverse data landscapes.