SALT data pipeline: an overview

The multi-instrument Southern African Large Telescope (SALT) currently has an instrument complement that includes the high-resolution spectrograph (HRS), the Near Infrared Washburn Astronomical Laboratories Spectrograph (NIRWALS), the Robert Stobie Spectrograph (RSS), and the imaging camera SALTICAM. The processing of the data obtained by the various instruments is controlled by the SALT data pipeline. Coded in Python, the pipeline runs daily and employs a workflow system to control the sequence of the data processing: (i) primary (basic) reductions applied to the data based on the requirements for each instrument, (ii) administrative database updates, (iii) secondary (science) reductions for HRS and RSS data by their respective science data pipelines, (iv) data distribution to users, and (v) internal tasks needed for the efficient management of the telescope science operations. The pipeline workflow system was developed to allow parallel processing within the workflows for the different stages of data processing. The stages in which parallel processing is applied are the primary reductions of the different instruments, the instrument secondary pipelines, the distribution of the processed data to different principal contacts, and the data quality monitoring process.