Challenges and Future Directions in Similarity Assessment of Big Data Analytics Workloads

Performance modeling can enhance the resource efficiency of clusters and distributed data processing applications, however, the modeling data available is frequently insufficient. Simultaneously, workloads regularly exhibit similar traits that could be leveraged to refine and enhance performance models for greater resilience. Numerous recent studies have made efforts in this direction by assessing the similarity of workloads in one way or another, although the effectiveness of the similarity assessment techniques employed for data exploitation remains questionable.This paper examines such techniques, investigating their underlying presumptions and common strategies for assessing similarity. We carry out a range of experiments in a public cloud environment using representative algorithms, datasets, and parameterizations to thoroughly showcase both the strengths and weaknesses of current methods in the discipline. This analysis serves as a basis to propose ideas for future research directions on workload similarity, with the ultimate goal of deriving informed resource configurations for big data analytics workloads.