Dataset Integrity Done with Git (didg) – A framework for managing neuroscience datasets along processing phases

Modern neuroscience, among others, produces increasing amounts of raw data, e.g. from multi-channel electrophysiology or functional imaging. These data often originate from individual experimental sessions, each resulting in one dataset that is self-contained in the sense that it includes all the information for subsequent processing. Thus, these datasets are independent except on the conceptual level imposed by the experiment. Processing in this context refers to each step in the analysis pipeline, automatic or manual, which can be performed on datasets prior to pooling or cross-referencing them. Examples include: spike detection and spike sorting, or extracting downsampled local field potentials in electrophysiology; in functional imaging, regions of interest are defined and fluorescent signals saved as simple time series. In many cases, the extracted relevant data is much smaller than what was originally recorded. Therefore, it is desirable to keep only the smaller and processed datasets for final analysis on a local computer, while the original raw data and intermediate datasets can be stored on suitable server infrastructure. Maintaining the integrity of datasets throughout this process, including the relationship across phases, is crucial for reproducibility of analysis workflows and becomes increasingly challenging with the amount of data. This is especially important for labs that do not have massive infrastructure or people managing it, and instead rely on whatever their institution provides, which could be as basic as a network file share. We propose a convention-driven framework to leverage well-established software tools, namely Git [1] and Git LFS [2] (large file storage) to solve the outlined challenges. The main idea is simple: Datasets in uniquely named directories are originally added into separate Git branches, following a customizable naming scheme, to allow independent retrieval and processing of each dataset. Git's history with a common, empty (no files) ancestor commit allows pooling of the fully-processed datasets using simple merge strategies, while maintaining a traceable record of each dataset's history. Leveraging Git LFS for storage and transfer of large files ensures binary data integrity and prevents several scenarios of accidental data loss. Furthermore, commands like “git lfs prune” allow to effortlessly free up storage space on local machines without the need to double check which data was successfully transferred previously, avoiding many human/organizational errors. We also aim at creating a new command line tool "didg", which is still under development, to facilitate the use of this framework, especially for users less familiar with Git. We aim to make didg a helpful and easy-to-use companion for neuroscience labs to manage the processing flow and integrity in a world of increasing data volumes. One of the main motivations of this kind of convention-driven framework is the independence of additional tools (besides Git, which should be in everyone’s use already). This has allowed us to freely combine and leverage the existing tools for data processing rather than replacing them, whether that’s commercial software or custom written scripts, and even integrate manual and semi-manual steps within the same traceable analysis pipeline, all while keeping local hardware and other requirements to a minimum.