SMAFIRA Shared Task at the BioNLP'2025 Workshop: Assessing the Similarity of the Research Goal

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14310%2F25%3A00143569" target="_blank" >RIV/00216224:14310/25:00143569 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2025.bionlp-1.33/" target="_blank" >https://aclanthology.org/2025.bionlp-1.33/</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.18653/v1/2025.bionlp-1.33" target="_blank" >10.18653/v1/2025.bionlp-1.33</a>

Result language
angličtina
Original language name
SMAFIRA Shared Task at the BioNLP'2025 Workshop: Assessing the Similarity of the Research Goal
Original language description
We organized the SMAFIRA Shared in the scope of the BioNLP'2025 Workshop. Given two articles, our goal was to collect annotations about the similarity of their research goal. The test sets consisted of a list of reference articles and their corresponding top 20 similar articles from PubMed. The task consisted in annotating the similar articles regarding the similarity of their research goal with respect to the one from the corresponding reference article. The assessment of the similarity was based on three labels: "similar", "uncertain", or "not similar". We released two batches of test sets: (a) a first batch of 25 reference articles for five diseases; and (b) a second batch of 80 reference articles for 16 diseases. We collected manual annotations from two teams (RCX and Bf3R) and automatic predictions from two large language models (GPT-4omini and Llama3.3). The preliminary evaluation showed a rather low agreement between the annotators, however, some pairs could potentially be part of a future dataset.
Czech name
—
Czech description
—

Project
—
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year
2025
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection
Proceedings of the 24th Workshop on Biomedical Language Processing
ISBN
9798891762756
ISSN
—
e-ISSN
—
Number of pages
8
Pages from-to
388-395
Publisher name
Association for Computational Linguistics
Place of publication
Stroudsburg
Event location
Vienna, AUSTRIA
Event date
Aug 1, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
001616252100033

Similar results(10)