DPQL: Applications for Holistic Data Profiling

Data profiling is the process of extracting implicit metadata, such as data types, attribute statistics, and various types of data dependencies, from raw datasets. Because structural metadata is often not stored explicitly, many data management and data engineering applications rely on data profiling to identify, for example, functional dependencies, inclusion dependencies, or uniqueness column combinations. The utilization of automatically discovered metadata within use cases, such as data discovery, integration, normalization, cleaning, or optimization, is, however, a still very complicated process. This is because existing data profiling approaches consider different types of metadata in isolation, although in practice, many use cases require specific combinations, i.e., patterns of structural metadata. For this reason, we recently proposed the Data Profiling Query Language DPQL that can express (and in the near future holistically discover) arbitrary patterns of various types of metadata. DPQL query patterns allow data scientists to express exactly what metadata real-world applications require. This eliminates otherwise complicated post-processing efforts, reduces result sizes, and might lead to significantly shorter profiling times. To demonstrate the expressiveness and versatility of DPQL, we survey a variety of data engineering applications in this paper and solve their metadata requirements with concrete DPQL query patterns. We also measure DPQL result sizes on two benchmark datasets and compare them to the result sizes of standard data profiling algorithms to demonstrate that holistic data profiling with DPQL produces not only more suitable, but also much smaller result sets.