Real-Time Automated Monitoring Architecture for Ensuring Quality ETL Processes in Healthcare Data: A Case Study from Cologne University Hospital’s Medical Data Integration Center

This case study discusses the effectiveness of implementing a real-time automated monitoring architecture using the ELK Stack (Elasticsearch, Logstash and Kibana) to ensure data ingestion quality within the Medical Data Integration Center (MeDIC) at the University Hospital Cologne. By streamlining the ETL (Extract, Transform, and Load) log analysis process, this system minimizes the need for manual effort and brings increased efficiency and precision in analyzing data quality issues in real-time, detecting errors and potential problems, including the ability to uncover new errors. Over a six-month period, the implemented dashboard was able to process the ingestion logs of millions of files to provide valuable insights for the stakeholders in the decision-making process.