An adaptable workflow for uniformly structured text data for text and data mining

Digital texts constitute the source data for many scientific studies that employ methods such as text and data mining (TDM), with some examples described in [1-5]. The major challenges for conducting such studies include the practical access to and aggregation of texts from different sources with non-uniform data formats and structures that complicate the application of TDM methods and render them less effective as well as legal hurdles. [6] To address these challenges, the University and State Library Darmstadt is developing an adaptable workflow in a project funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) [7] to transfer documents from different sources into a standardised XML format based on the specifications of the Text Encoding Initiative (TEI) [8]. The workflow starts with document selection, which can be based on specific acquisition requests (just in time) or on the library's collection activities based on its profile (just in case). This is followed by the rights clearance and actual access to the documents for the library – ideally including media elements in addition to full texts in XML and PDF formats. In cases that have already been successfully negotiated, access is granted by, for example, delivering data to an SFTP server provided by the library or by retrieving data from a publisher's server. The documents are then sorted based on their Crossref metadata and an initial validation of the XML documents is conducted. The focus here is not on exact adherence to standards, but on ensuring that the documents can be automatically converted at a later stage. The data is then stored in a long-term archive in an Archivematica [9] system. Following the OAIS model (Open Archival Information System, see [10]), the data is subsequently transferred from a Dissemination Information Package (DIP) into an initial delivery system: a DSpace repository primarily used to store and provide access to a wide range of data formats making use of a powerful rights and role management if necessary. From this system, the conversion of the full text into the XML target format is conducted using XSLT scripts. The text is stored in an eXist XML database as second delivery system. The full texts can be searched and accessed through the APIs of the delivery systems. The collected literature will be listed in the hebis union catalogue [11]. This general workflow and the here established methods are used in Measure S-7 of the NFDI4ING consortium to create and provide a corpus of documents that are relevant for the engineering sciences to be used in text and data mining. During the current funding period (2023-2025), the project focuses on open access literature that is provided by publishers in a structured format, typically JATS-XML or BITS-XML. An extension to licensed literature is planned for later project phases, as is the creation of the structured text data in the target format from unstructured analog or digital source data. In the future, the workflow will be expanded to allow a joined content collection with split labor by different scientific libraries.