Event Driven Architecture for the IMAP Science Data Center

The new technologies and techniques within the cloud have provided a huge variety of new tools for data pipelines and data management. This presentation explores the tools and techniques used by the IMAP SDC at CU Boulder/LASP to create a flexible, reliable, and efficient event-based processing pipeline. Using tools like AWS, infrastructure as code, and Docker, the IMAP SDC has created data storage, tracking, and processing capabilities for the 10 instruments on IMAP, many of which are interconnected within processing. By creating an event based system, the SDC is capable of extending and modifying processing based on changing requirements, processing in parallel on multiple Docker containers, and ensuring that files that arrive at unpredictable times are always processed as quickly as possible.