The Utility of PIDs in Harvesting Open Data Repositories: Challenges in Operating a National Data Discovery Service

Lunaris is a national data discovery service for Canada. Providing complete and structured, preferably standards-based, metadata for data records enables Lunaris and other discovery services to effectively find potentially relevant datasets, filter them to concretely identify Canadian datasets, and crosswalk them to an internal metadata schema. We will focus on practical challenges related to our mandate of harvesting Canadian datasets, but our recommendations will be applicable to discovery of other subsets of research data at large. The benefits of persistent identifiers (PIDs) are often discussed in an abstract or aspirational way. Lunaris' experience harvesting repositories reveals concrete scenarios in which PIDs are critical to effectively handling repository metadata. This presentation will outline our procedure for harvesting a new repository and highlight situations in which repositories' use of PIDs and other structured metadata improve this procedure, making explicit recommendations for structured metadata use. We will then discuss the way specific PIDs and other metadata elements enable us to help our users effectively discover records in this cross-repository environment. Finally, we will discuss future work in handling emerging PIDs and metadata standards.