Preparing a dataset for new life: cleaning a longitudinal study for new analyses and linkages

Abstract Issue The Belfast Youth Development Study (BYDS) was conducted from 2000-2010, comprising 7 waves of collection. The data contain valuable insights into the health and social lives of adolescents but were stored using inconsistent management practices and lacked a metadata crosswalk describing all variables and their content. Primary challenges to new analysts included shared questions entered using varying notations across waves of collection (“mother” and “yes” in a “mother lives at home” variable), single question responses stored wide across multiple variables, and a need for comprehensive metadata. Description Individual questions were un-joined from the linked dataset, cleaned using the tidyverse suite of packages in R, and then left-joined to the primary dataset. Where responses were recorded differently across years, scales were standardised across waves of collection. Questions were arranged in their original order to maintain dataset integrity. Simultaneously, a crosswalk was created of each question, its old and new variable names, and its response scales. Rows for old variable names and response scales were collapsed into comma-separated values and deduplicated by the new variable names using the dplyr package. Results Following this work, the BYDS dataset is ready for new researchers to analyse and describe findings. A large grant has recently been awarded to conduct a new wave of data collection, the analysis of which will be enhanced by the cleaned dataset and its associated metadata. Lessons It is feasible and desirable to maintain older datasets through semi-automated practices facilitated through the tidyverse suite of packages in R. Improvements in data management since the initial digitisation of large datasets more than 20 years ago provide a framework for conducting this work. Key messages • The tidyverse suite of packages allow for low-burden cleaning and maintenance of important historical datasets to facilitate their ongoing use and preservation. • There is value in supporting staff training on simple data management skills in R to support lower burden in data management practices.