Pitschi: Rolling Out an Imaging Big Dataset Management Framework for Advanced Microscopy

Pitschi (Particle Imaging depoT using Storage CacHing Infrastructure) is a state-of-the-art structured repository designed for managing large electron and correlative microscopy datasets at the Centre for Microscopy and Microanalysis (CMM) at UQ. Developed under the ARDC Australian Characterisation Commons at Scale project, Pitschi is built on the open-source Clowder data management framework . It adheres to FAIR data principles and fully integrates with UQ's instrument booking system and storage infrastructure. Pitschi provides end-to-end data management, encompassing raw data capture, storage, ingestion, and indexing, with automatic metadata extraction from compatible file formats, thereby improving searchability and discovery capabilities. The ingested data are accessible across various platforms, including HPCs, personal computers, and processing platforms, with seamless data transport managed by the Metropolitan Data Caching Infrastructure (MeDICI). Since its deployment in 2022, Pitschi has been rolled out across all CMM laboratories and is operational on 40 scientific acquisition beam-lines. Enhancements include expanded data synchronization capabilities to support a broader range of instruments and onboarding for commercial users and new third-party laboratories. Significant successes include widespread user appreciation for the system's "drop and forget" data transfer capability. However, challenges remain, particularly regarding the underuse of advanced search and the complexities of user support across multiple sites. We are planning substantial infrastructure and application upgrades to enhance Pitschi's capabilities, aiming to improve user experience, expand functionality, and ensure robust support across all sites. This update highlights our commitment to advancing data management in microscopy, driving greater efficiencies, and supporting the research community with reliable, cutting-edge tools.