ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science
ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.
Authors
- Elliott Slaughter (ORCID: https://orcid.org/0000-0001-9725-1305)
- Aaron S. Brewster (ORCID: https://orcid.org/0000-0002-0908-7822)
- Antoine Dujardin (ORCID: https://orcid.org/0000-0001-9810-1341)
- Jeffrey J. Donatelli (ORCID: https://orcid.org/0009-0003-7173-0174)
- Billy K. Poon (ORCID: https://orcid.org/0000-0001-9633-6067)
- Johannes Blaschke (ORCID: https://orcid.org/0000-0002-6024-3990)
- Monarin Uervirojnangkoorn
- I.D. Young (ORCID: https://orcid.org/0000-0003-4713-9504)
- W. Kroeger
- Seema Mirchandaney
- Chun Hong Yoon (ORCID: https://orcid.org/0000-0003-1482-1662)
- Vinay Ramakrishnaiah (ORCID: https://orcid.org/0000-0002-2925-4798)
- Wu-chun Feng (ORCID: https://orcid.org/0000-0002-6015-0727)
- Peter McCorquodale
- Vidya Ganapati (ORCID: https://orcid.org/0000-0003-2597-5640)
- Frédéric Poitevin (ORCID: https://orcid.org/0000-0002-3181-8652)
- Nicholas K. Sauter (ORCID: https://orcid.org/0000-0003-2786-6552)
- Christine Sweeney (ORCID: https://orcid.org/0000-0003-2352-490X)
- Felix Wittwer (ORCID: https://orcid.org/0000-0002-5050-1669)
- Daniel Tchoń (ORCID: https://orcid.org/0000-0003-4798-867X)
- Daniel W. Paley⧓ (ORCID: https://orcid.org/0000-0003-1161-5142)
- Robert Bolotovsky
- Christopher O’Grady
- Derek Mendez (ORCID: https://orcid.org/0000-0002-2269-0060)
- Amedeo Perazzo
- Michael E. Wall
- Niteya Shah
Institutions
- Los Alamos National Laboratory (US)
- Lawrence Berkeley National Laboratory (US)
- SLAC National Accelerator Laboratory (US)
- Research Applications (United States) (US)
- National Energy Research Scientific Computing Center (US)
- Virginia Tech (US)
- Numerical Algorithms Group (United Kingdom) (GB)
Publication Details
- Journal
- Frontiers in High Performance Computing
- Published
- 2024-10-03
- DOI
- https://doi.org/10.3389/fhpcp.2024.1414569
- Citations
- 2
- Primary Topic
- Advanced X-ray Imaging Techniques
- Type
- article
- Field-Weighted Citation Impact
- 1.22
Funders
- U.S. Department of Energy
- National Energy Research Scientific Computing Center
- National Institutes of Health
- Office of Science
- National Nuclear Security Administration
- National Institute of General Medical Sciences
- Basic Energy Sciences
- Advanced Scientific Computing Research
- Biological and Environmental Research
- Oak Ridge National Laboratory
- SLAC National Accelerator Laboratory