KIEBIDS: A Modular Framework for Scalable Information Extraction from Written Records in Natural History Collections

Natural history collections preserve invaluable records of biodiversity across time and space. Each specimen is typically accompanied by one or more labels documenting provenance, locality, and contextual data. Mobilizing this information is crucial for research on biodiversity change, biogeography, and taxonomy. To date, much of this data remains inaccessible for computational knowledge engineering approaches because manually processing and converting the sources into structured, interoperable data formats is a labor-intensive challenge due to the volume, heterogeneity, and complexity of the documents—and curatorial resources for fulfilling these tasks are generally insufficient. Artificial Intelligence (AI)-based methodologies can significantly accelerate this process and make it economically scalable. We present KIEBIDS*1, an open-source framework for specifying and executing AI-based workflows for information extraction from specimen label images. Following a linear data-pipeline architecture, workflows comprise five sequential functional steps for information extraction: image pre-processing (to prepare input images for subsequent analysis), layout analysis (to identify image regions that are relevant for information extraction), optical character recognition (to identify text on the syntactical level), semantic parsing (to identify text that references categories of interest), entity linking (to identify entities of interest mentioned in the text with authority records). image pre-processing (to prepare input images for subsequent analysis), layout analysis (to identify image regions that are relevant for information extraction), optical character recognition (to identify text on the syntactical level), semantic parsing (to identify text that references categories of interest), entity linking (to identify entities of interest mentioned in the text with authority records). Modularity and adaptability are central design principles for the framework's architecture. Each function can be realized by one or more modules that operate independently through file-based input and output, enabling substitution or extension as new technologies emerge. This ensures flexible adaptation to various information extraction goals or new data domains. In the current release, image pre-processing is implemented using the OpenCV framework with steps for resizing, grayscale conversion, noise reduction, and binarization. Layout analysis, based on the Segment Anything Model, identifies image regions that depict labels. Character recognition is implemented using two alternative modules. Besides EasyOCR, Moondream is used to leverage locally-deployable vision-language model (VLM) technology. Semantic parsing is implemented using spaCy and regular expressions, as rule-based parsing has proven efficient for syntactically well-defined entities, such as dates or coordinates, given the sparse context of label texts. Entity linking, in the current release, is realized for geographical place names using the GeoNames application programming interface (API). The input for a given workflow run consists of document images and configuration parameters. The configuration parameters encompass settings for the pipeline as a whole, such as location of input and output files and execution mode, the configurable settings for each functional step of the pipeline, e.g., models to be used, model parameters or the tag selection for the semantic tagging. settings for the pipeline as a whole, such as location of input and output files and execution mode, the configurable settings for each functional step of the pipeline, e.g., models to be used, model parameters or the tag selection for the semantic tagging. The workflow's output are PAGE-XML files containing image annotations, including the extracted and annotated text. Optionally, intermediate data and evaluation metrics can be assessed. Integrating seamlessly with Python codebases, Prefect is used for scheduling, monitoring, and graphical user interaction. By combining existing open frameworks rather than developing new components, the project leverages recent advances in computer vision and natural language processing to mobilize biodiversity data. Future developments will focus on improving user experience, integrating better models for handwritten text, and expanding semantic analysis capabilities. KIEBIDS' source code*2 is openly available and locally deployable with moderate hardware requirements.