Lessons learned refactoring the EuXFEL’s central data acquisition system

The European XFEL, a world-leading X-ray light source, boasts high brilliance and fast burst repetition rates supported by bespoke MHz-imaging detectors, resulting in data rates close to 20 GB/s. Correlating auxiliary data from various sources with these images requires a centralized Data Acquisition (DAQ) system that ingests data from Karabo, and outputs aggregated data into a common HDF5 format. Facility-side calibration and processing rely on the stability of this data format. We discuss refactoring this major, mission-critical software to scale for future data reduction needs. The system transitioned from a blackboard to a pipelined design modeling unidirectional dataflows. Refactoring was prepared with extensive tests for verification. Unit tests, often following test-driven development, accompanied refactoring and new features. A high continuous integration coverage is enhanced by system-level tests. The refactored system was introduced incrementally over 3 months, following a „test early, fail early“ philosophy. The new system successfully implements critical petabyte-scale data reduction, and enhanced status monitoring capabilities.