MDSplusML—Optimizations for data access to facilitate machine learning pipelines

The MDSplus data management system is widely used in the magnetic fusion energy research community for data storage, management, and remote access. The system provides data access through a vector based, interpreter API. It was developed and optimized for rapid single shot analyses. Machine Learning applications require data from large numbers of shots and potentially from different experimental devices. We are developing tools to enable the rapid retrieval of limited sets of data from large numbers of shots. The system will cache the requested quantities in a data warehouse overnight, and be able to quickly provide them as inputs to machine learning tasks. The cache will eventually be both transparent and extensible. At this time, various caching mechanisms are being tested and benchmarked using the queries for approximately 100 quantities that are typically used by disruption-warning ML workflows. The performance of various caching schemes varies greatly depending on the environment they are deployed in. We provide comparisons of the performance of native MDSplus, HSDS cache, and mongodb cache in various environments. The end goal is to provide fast data access to commonly queried quantities regardless of the environment.