L'intégrité des flux de données

Data quality is crucial in modern data management, ensuring accuracy, consistency, completeness, and reliability. While static databases handle integrity well, streaming environments require real-time validation to prevent inaccuracies from propagating. This thesis explores streaming integrity, defining it as adherence to predefined constraints. In real-time applications like financial trading and electricity monitoring, anomalies can indicate malfunctions or proper activation of automated backup systems. Addressing such cases requires detecting and annotating inconsistencies rather than modifying or discarding erroneous data. The thesis presents three key contributions: InkStream for evaluating continuous queries with inconsistency annotations, a semiring-based approach for continuous query provenance, and a study on the Event Processing Language (EPL). InkStream introduces a framework for annotating inconsistencies in continuous queries, adapting provenance methods to dynamic streaming environments through polynomial semirings. The approach integrates window-specific modifications to ensure computational feasibility. A key innovation is the Consistency Graph Summary, optimizing annotation propagation using transitivity properties of binary constraints, reducing complexity. A streaming-positive algebra extends operations to support consistency annotations in relational streams. Evaluations show up to 80% performance improvement and minimal runtime overhead. The thesis also introduces When-Provenance, extending provenance models with time intervals for tracking data derivations. Window operators divide infinite streams into finite segments. Selecting optimal window sizes is challenging, as small windows omit relevant data, while large ones increase memory usage. The framework employs interval-based semirings, leveraging balanced binary search trees and 2D orthogonal range counting for rapid query recomputation, enhancing debugging and auditing. The EPL study formalizes its data model, focusing on hierarchical, object-based entities essential for event stream processing. A systematic study identifies ambiguities and inefficiencies, such as the NOT operator’s ambiguous behavior and redundancy issues with the EVERY construct. A modified grammar improves query optimization techniques. The research follows the Macro-Meso-Micro framework, structuring research questions into broad, refined, and specific levels. A key aspect is the Continuous Violation Mapping problem: how do input constraint violations affect query results? A commuting diagram approach visualizes integrity violation propagation, showing the limitations of traditional repair techniques and advocating for consistency-aware querying. The thesis reviews related work in database integrity management, streaming data repair, and provenance tracking. It evaluates methodologies through qualitative and performance studies, an InkStream demonstration, and use cases in financial analytics and review bombing detection. Findings suggest extending the provenance model to additional constraints and languages, exploring alternative data models, and investigating adaptive windowing techniques. This thesis lays the foundation for integrity-aware stream processing, introducing techniques for consistency tracking without disrupting real-time analytics. The proposed methods impact industries reliant on high-quality streaming data, including finance, healthcare, and IoT applications.