Joint Named Entity and Relation Extraction from Software Mentions in Scholarly Articles

Software plays an essential role in scientific research. However, software is usually not cited formally in academic documents, resulting in various informal mentions across research documents. Automatic identification and disambiguation of software mentions and related attributes such as URLs, Versions, License, Citation, and other metadata contribute to the better understanding, accessibility, and reproducibility of research, but it is a challenging task. Software Mention Detection (SOMD2025) is a community-driven shared task focused solely on extracting software mentions from scholarly articles. Aligned with NFDI4DS's broader mission, it addresses the information-extraction step in the research-data lifecycle. SOMD2025 is the second iteration of the SOMD2024 shared task. SOMD2024 was a part of the 2024 Natural Scientific Language Processing Workshop, focused on three subtasks related to information extraction: Software Mention Detection, Additional Attribute Detection, and Relation Classification, between given software and attributes. This year's task builds on the success of the preceding shared task, focusing on the joint learning of three collective tasks. This shared task challenges participants to jointly learn the automatic extraction of software mentions, its attributes, and their relations from scholarly documents. It is co-located with the Scholarly Document Processing (SDP) workshop, ACL 2025, in Vienna, Austria, in August 2025. The SOMD2025 shared task comprises two distinct phases. Phase I is aimed at model development, where participants construct machine learning models capable of automatically identifying software mentions and their attributes and classifying their relationships. This phase utilizes a gold standard training set and a test set assessing the effectiveness and accuracy of proposed models. Phase II encourages participants to apply their models from Phase I on an out-of-distribution dataset comprised of scholarly documents that were not part of the training or test set used in Phase I. This phase aims to evaluate the proposed models in terms of their generalizability and robustness. The competition (March-April 2025) attracted six distinct teams and lasted for two months. Four teams entered Phase I, while five participated in Phase II, with three teams participating in both phases. The joint evaluation metric is computed as the average of the F1 scores for software and attribute mention extraction and the F1 score for relation classification between the extracted entities. The winning team achieved an F1 score of 0.89 in Phase I and 0.63 in Phase II. This competition has effectively improved the F1 scores, surpassing the baseline scores of 0.804 for Phase I and 0.491 for Phase II. We had valuable insights into the performance difference in the two phases, showcasing a bias of the model's reproducibility in out-of-distribution datasets. When given the unseen context, models generally struggle to perform as well as in a familiar context. This provides directions for further work on the robustness and generalizability of a model trained on specific data. Moreover, we have started an open submission phase, inviting researchers to submit their results on the benchmark dataset from both phases. This final submission phase is not part of the competition but an initiative encouraging ongoing collaboration and facilitating long-term engagement within the research community.