bwSFS2 – a Storage Infrastructure Service to Manage and Document Large Scale Research Data

The purpose of bwSFS-2 is to provide an infrastructure to store and document large scale research data at the University of Stuttgart and University of Hohenheim. It extends Baden-Württemberg's "Storage For Science" service, initiated by the Universities of Freiburg and Tübingen. Composed of storage hardware and management software, developed collaboratively with Nodeum, bwSFS-2 meets the needs of researchers, data stewards, and system administrators (see Figure 1). The solution enables the use of customizable metadata templates to document datasets in a structured fashion, in line with the FAIR principles introduced by [1]. Whilst other infrastructures focus on discipline-specific workflows or data-types, like OMERO for microscopy images, bwSFS-2 is designed as versatile, domain-agnostic solution for storing and describing research data. Compared to Coscine [2], which also enables structured metadata for research data using individual metadata profiles, bwSFS-2 focuses on data economy and quality. The metadata itself can be provided via web-application or by putting structured metadata files in a JSON format in the file system together with the primary data. All metadata is indexed and findable over the web-application, but can also be exported back to the filesystem to keep the metadata with the data for archiving or exporting datasets. In bwSFS-2, the data are organized in terms of dataset(s) – a collection of related files and folders of research data – which can be described by metadata. Beyond process metadata such as methods, tools and parameters (see Figure 3), relations between data sets (managed within bwSFS-2) can be captured to describe provenance (see Figure 2). The permissions to access the data (read-only or read-write) can be managed by the researchers for their data sets through the bwSFS-2 web application. Built on three different storage layers (file service via SMB/NFS3, S3 and tape), bwSFS-2 implements data economy by allowing to move data from the (expensive) file service (semi-)automatically to more cost-efficient storage layers (S3-object storage or tape), when not frequently used. Consequently, a dataset is technically represented by a network-share (file service), a S3 bucket (object storage) or an archive stored on tape, depending on the storage location. The datamover (service) ensures that access permissions are maintained regardless of the storage layer.1 An important feature of the bwSFS-2 software is the separation of infrastructure management functions (e.g., resource creation) and the underlying storage infrastructure(s) by so-called connectors2, which prevents vendor lock-ins and allows – in principle – to run the software with different storage hardware. (Semi-)automated checks help assessing the (metadata) quality of the datasets. Data is stored only for a limited time and has to be published or archived to be kept (see figure 3). In that sense, bwSFS-2 helps to manage research data over the entire research data lifecycle including the publication of well-documented data that can be made available via the NFDI. The software component(s) of bwSFS-2 are currently developed and will be published open-source. The service bwSFS-2 is expected to go into production in late 2025.