Going Deeper: Advancing Data Discovery Through Content Analysis

Efficient discovery of relevant data for scientific applications is a major challenge for researchers. This is particularly compounded for general purpose data repositories that store heterogeneous datasets with diverse metadata for different data types. The Environmental Systems Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) is a U.S. Department of Energy repository that archives Earth sciences data. The repository recommends archiving data in specific data/metadata community standards to improve data discovery and reuse. While the repository search enables discovering data based on textual metadata provided by users, there was a need to discover data based on the file contents given the vast numbers of files archived within the datasets. To address this issue, we developed a fusion database (FusionDB) that enables a deeper search of the metadata within the files of each dataset. FusionDB transforms this by validating datasets against community standards and indexing the actual parameters within each file, like location coordinates, time ranges, and measurement types, making them searchable and filterable. The core engineering insight was recognizing that metadata tells you what researchers think their data contains, but automated content analysis reveals what it actually contains. The results demonstrate the power of thoughtful automation: review processes have been significantly expedited, data quality improved through automated feedback, and researchers can now preview and filter datasets based on the file contents before download. This approach will create a positive feedback loop where better tooling encourages better data practices, with the potential to raise standards across the entire community. For fellow RSEs, FusionDB demonstrates how solving one community's pain point can catalyze systemic change. By automating the tedious work, we didn't just save time, we enabled researchers to focus on science while positioning the community for improved data quality practices. The lesson: sometimes the most impactful software engineering isn't about building new capabilities, but about removing friction from existing workflows at exactly the right point in the process.