RACCOONS: Data Management Library for High-Performance Astrometric Reduction Pipelines

An astrometric reduction pipeline is a sequence of steps for calculating spatial positions and velocities of stars from their angular positions at different times on the celestial sphere as measured by telescopes. The accuracy of the results depends on the number of observations per star, because even the finest telescopes are afflicted with various noises. To reach scientifically valuable accuracy, the total number of observations undergoing reduction exceeds billions, demanding highly efficient Big Data management tools to support calculations. In this work, we introduce the RACCOONS library, which provides high-performance lock-free methods for handling immutable and mutable data. It is written in C++, has a Python binding, and implements efficient mechanisms for querying the observational data of a certain astrometric mission, calculating statistical properties of requested datasets, and acting as a large in-memory storage for mutable data by joining together the RAM of several cluster nodes, utilising the remote direct memory access. This library is currently reused in three software systems within the astrometric reduction pipeline for the Japan Astrometry Satellite Mission for Infrared Exploration (JASMINE): direct and iterative astrometric solvers, and a solution visual analytics system. At the current research and development step, we tested RACCOONS and the corresponding software on relatively small CPU clusters with 0.2 PFLOPS capacity and proved good scalability of both calculations and data treatment. The next step for future work is to deploy RACCOONS-powered software on a 2 PFLOPS cluster.