SOMD2025: A Challenging Shared Tasks for Software Related Information Extraction

The use of software in acquiring, analyzing, and interpreting research data underscores its role as an essential artifact of scientific inquiry.Understanding and tracing the provenance of software in research helps in reproducible and collaborative research works.In this paper, we present an overview of our second iteration of the Software Mention Detection (SOMD) shared task as a part of the Scholarly Document Processing (SDP) workshop, that will be held in conjunction with ACL in 2025.The objective of this shared task is to encourage participants to reevaluate the methodologies employed in the tasks of joint named entity recognition (NER) and relation extraction (RE) for software mentions using the gold standard benchmark that has been provided.Our shared task has two phases of challenges.First, the participants focus on implementing a joint framework for NER and RE for the given dataset.Furthermore, the second phase encompasses an out-of-distribution dataset, which is utilized to assess the generalizability of the methodologies proposed in Phase I.The competition, which transpired from March to April of 2025, garnered the participation of 18 individuals and spanned a duration of two months.Four teams have finished the competition and submitted full system descriptions.Participants applied various approaches, including joint and pipeline models, and explored data augmentation with LLM-generated samples.The evaluation was based on a macro F1 score for both NER and RE, with the average reported as the SOMD score.The winning teams achieved a SOMD score of 0.89 in Phase I and 0.63 in Phase II, demonstrating the challenge of generalization.