Building the Princeton Data Commons

In this presentation we will talk about how we are building the Princeton Data Commons, a set of products and services to help the university library manage research data at Princeton University. The focus of this presentation will be on how we are building the software projects that underlie these services, in particular we'll talk about how we developed our data curation tool (PDC Describe) and our discovery tool (PDC Discovery) but the presentation will include not only the technical aspects of building software but also the social components that are part of designing, building, releasing, and maintaining infrastructure software projects for the long haul. In building the Princeton Data Commons (PDC) we took the unconventional approach of building a new set of software tools from scratch rather than using an out of the box solution. To do this we decided to break apart the research data infrastructure problem [1] into smaller components (e.g. curation, discovery, storage) that we can tackle independently and with a variety of approaches. As Trevor Owens said "the recurring idea of the singular technological super system that 'solves the problem' is a distraction not worth chasing" [2] and with that in mind we are developing different tools for different parts of the problem, and we are also leveraging existing commercial and open-source tools where appropriate. Building infrastructure projects is rather a linear and straightforward process. We have embraced the iterative and incremental approach suggested by Agile software methodologies [3] and developed our software following this model. This means that we try to prioritize early delivery of features over their completeness and continue to evolve our software products after their initial release. Releasing software as minimal viable products is not always easy or without controversy but we have found that by continually enhancing our products rather than keep delaying their release until all the features have been completed allows for a better feedback loop from users. We will also talk about the challenges of developing research data infrastructure and how we are working through them, particularly as the size and complexity of datasets continues to grow at a rapid pace. We will also discuss some of the new features and enhancements that we have planned for future releases.