XPCS use case in DAPHNE4NFDI - Data analysis workflow and infrastructure

X-ray Photon Correlation Spectroscopy (XPCS) is an experiment technique apt to study dy-namic processes in samples of hard and soft condensed matter, exploiting the beam coher-ence features offered by photon-sources like X-ray Free-Electron Lasers (XFELs) and syn-chrotrons. The XPCS use case developed in the DAPHNE4NFDI consortium and presented here covers workflows of processing and analyzing experiment data that stems from the in-teraction of the spatially highly coherent photon pulses at the Materials Imaging and Dynamics (MID) instrument[1] of the European XFEL with protein molecules in solution and detection of diffraction speckle fluctuations with a MHz repetition rate adaptive gain integrating pixel detec-tor (AGIPD). The very specialized experiment setup of MHz-XPCS[2] – combining small-angle X-ray scat-tering (SAXS) configuration of the AGIPD detector with the exploitation of European XFEL pulse-train patterns[3] in order to probe the diffusive dynamics of biological samples with sub-microsecond temporal resolution – has evolved to become a standardized and (somewhat) routine protocol. Treatment of the raw image data, on the other hand, remains a challenge, in particular due to the vast data volume[4] combined with a highly complex data structure result-ing from multiple aligned instrumentation sources. Maturation of data correction and analysis methods as well as the software operation interfaces has the clear aim of standardizing and as much as possible automating the data processing workflow. Minimizing human intervention and maximizing throughput has the goal of keeping the rate of reduced data production close to the raw data acquisition rate while ensuring that the data quality is sufficient for meaningful interpretation and scientifically valuable results. Here we present the implementation of MHz-XPCS-specific algorithms to data analysis pipe-lines including parallel processing solutions and their required hardware.[5] Furthermore, we describe the data and metadata extraction tool DAMNIT, which takes user-defined experiment variables of interest, as well as descriptions of data processing steps, in high-level Python code and runs such workflows on automatic triggers whenever files belonging to a data-taking run are available. DAMNIT comes with a GUI for tabular view of variables vs. runs and versa-tile visualization features. We demonstrate how MHz-XPCS data pipelines are embedded to DAMNIT, thus getting closer to the goal of data analysis standardization and automation. We also discuss how DAMNIT can be connected with the DAPHNE4NFDI XPCS platform for pro-vision of open data to the XPCS community. [1] A. Madsen et al., "Materials Imaging and Dynamics (MID) instrument at the European X-ray Free-Electron Laser Facility", J. Synchrotron Rad., vol.28, no.2, pp. 637-649, Mar. 2021, DOI: https://doi.org/10.1107/S1600577521001302 [2] F. Lehmkühler et al., "From Femtoseconds to Hours – Measuring Dynamics over 18 Or-ders of Magnitude with Coherent X-rays", Appl. Sci., vol.11, no.13, pp. 6179-6211, Jul. 2021, DOI: https://doi.org/10.3390/app11136179 [3] W. Decking et al., "A MHz-repetition-rate hard X-ray free-electron laser driven by a su-perconducting linear accelerator", Nat. Photonics, vol.14, no.6, pp. 391-397, May 2020, DOI: https://doi.org/10.1038/s41566-020-0607-z [4] E. Sobolev et al., "Data reduction activities at European XFEL: early results", Front. Phys., vol.12, p. 1331329, Feb. 2024, DOI: https://doi.org/10.3389/fphy.2024.1331329 [5] A. Leonau et al., "A pipeline for Megahertz X-ray Photon Correlation Spectroscopy on soft matter samples at the MID instrument of European XFEL", to be submitted to J. Synchrotr. Rad.