ESRF's software strategy for high-throughput detectors: scalable data acquisition with real-time processing

The ESRF EBS upgrade dramatically increased the X-ray photon flux for the beamlines, thereby opening up new experimental opportunities but also requiring faster and more advanced 2D detectors. At ESRF, detectors are integrated to LIMA, a framework that provides unified control and acquisition Tango APIs for our control software, BLISS. To meet the challenges of new 2D detectors, a comprehensive data strategy has been developed. This includes a redesigned data acquisition framework, LIMA2, which aims to support an increasing data throughput with a scalable design, facilitate online data reduction (ODR) tasks and provide data access for Online Data Analysis (ODA) software. In addition, new detectors introduce new data models, such a multi-band, sparse, sub-byte or event-based data which need to be handled appropriately. In the framework of our data strategy, we give an update on the recent developments of the LIMA2 project components: the core library architecture and the hardware plug-ins for Dectris, Rigaku XSPA-1M, ESRF Smartpix detectors. We also showcase the processing plugin model illustrated by the serial macro-crystallography (SMX), photon correlation spectroscopy (XPCS) techniques. We demonstrate LIMA2's new capabilities with real-world applications on ID10 and ID15a/b and present the results. We conclude with our vision on future data models and their integration with the control, visualisation and processing softwares.