SMAFIRA Shared Task at the BioNLP’2025 Workshop: Assessing the Similarity of the Research Goal

We organized the SMAFIRA Shared in the scope of the BioNLP'2025 Workshop.Given two articles, our goal was to collect annotations about the similarity of their research goal.The test sets consisted of a list of reference articles and their corresponding top 20 similar articles from PubMed.The task consisted in annotating the similar articles regarding the similarity of their research goal with respect to the one from the corresponding reference article.The assessment of the similarity was based on three labels: "similar", "uncertain", or "not similar".We released two batches of test sets: (a) a first batch of 25 reference articles for five diseases; and (b) a second batch of 80 reference articles for 16 diseases.We collected manual annotations from two teams (RCX and Bf3R) and automatic predictions from two large language models (GPT-4omini and Llama3.3).The preliminary evaluation showed a rather low agreement between the annotators, however, some pairs could potentially be part of a future dataset.