Content-Based Dataset Retrieval Methods: Reproducibility of the ACORDAR Test Collection

The FAIR principles constitute a cornerstone of contemporary scientific methodology, with the Digital Library (DL) communityactively participating and providing significant advancements within thisframework. By taking a reproducibility approach, this paper centers onfindability, a pivotal aspect of scientific data management and stewardship. Specifically, we delve into the critical role of Data Search in enablingefficient retrieval across various contexts, including scholarly publicationsand scientific data management. Consequently, the convergence of DigitalLibrary and Information Retrieval (IR) domains underscores the necessity to adapt document-level IR techniques to optimize dataset retrievalprocesses.Dataset retrieval relies on dataset descriptions, hampered by incompleteand inconsistent metadata issues. Lately, there has been a growing emphasis on Content-Based Dataset Retrieval (CBDR), where metadataand dataset content are equally considered during indexing and retrieval.ACORDAR is the first open test collection to evaluate CBDR methods.It offered early insights into the benefits of integrating dataset contentin retrieval.Our study thoroughly assesses ACORDAR’s quality and reusability whileinvestigating the reproducibility of retrieval results. Concerns arise aboutaccessibility to the collection’s content due to broken links for 17.6 ofdatasets. Despite some errors and requiring non-trivial pre-processingsteps, we replicated most but not all CBDR methods, thus raising someconcerns about the suitability of ACORDAR as a reference test collectionto further advance CBDR research and to employ these methods in thecontext of DL