Dynamic stream clustering for real-time schema profiling with DSC+

Abstract Streaming applications often involve several sources that continuously produce data in diverse formats—for instance, IoT applications may receive heterogeneous messages from different devices. In such a scenario, technicians are interested in characterizing the incoming traffic to understand which kinds of data are flowing into the stream; this activity is called schema profiling and consists of producing key insights about the schema of data in a high-variety context. In this paper, we present a streaming approach to schema profiling called DSC+ (Dynamic Stream Clustering). The approach works under the overlapping sliding window paradigm and produces profiles by clustering the schemas extracted from the data according to the k-means algorithm. DSC+ is a two-phase algorithm: schemas are pre-aggregated into a coreset using a compact data structure called schema features and then clustered. Unlike previous approaches, DSC+ is designed to work in a dynamic and unknown domain, where the number of schema attributes is variable and is not known a priori. DSC+ implements rules to dynamically and efficiently update the profile in real-time, minimizing the computations at every slide of the window while maintaining consistency with the principles of k-means. Experiments are carried out on both real and synthetic datasets to demonstrate the efficiency and effectiveness of our proposal against related state-of-the-art algorithms.