SMAFIRA Shared Task at the BioNLP'2025 Workshop: Assessing the Similarity of the Research Goal

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14310%2F25%3A00143569" target="_blank" >RIV/00216224:14310/25:00143569 - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2025.bionlp-1.33/" target="_blank" >https://aclanthology.org/2025.bionlp-1.33/</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2025.bionlp-1.33" target="_blank" >10.18653/v1/2025.bionlp-1.33</a>

Jazyk výsledku

angličtina

Název v původním jazyce

SMAFIRA Shared Task at the BioNLP'2025 Workshop: Assessing the Similarity of the Research Goal

Popis výsledku v původním jazyce

We organized the SMAFIRA Shared in the scope of the BioNLP'2025 Workshop. Given two articles, our goal was to collect annotations about the similarity of their research goal. The test sets consisted of a list of reference articles and their corresponding top 20 similar articles from PubMed. The task consisted in annotating the similar articles regarding the similarity of their research goal with respect to the one from the corresponding reference article. The assessment of the similarity was based on three labels: "similar", "uncertain", or "not similar". We released two batches of test sets: (a) a first batch of 25 reference articles for five diseases; and (b) a second batch of 80 reference articles for 16 diseases. We collected manual annotations from two teams (RCX and Bf3R) and automatic predictions from two large language models (GPT-4omini and Llama3.3). The preliminary evaluation showed a rather low agreement between the annotators, however, some pairs could potentially be part of a future dataset.

Název v anglickém jazyce

SMAFIRA Shared Task at the BioNLP'2025 Workshop: Assessing the Similarity of the Research Goal

Popis výsledku anglicky

We organized the SMAFIRA Shared in the scope of the BioNLP'2025 Workshop. Given two articles, our goal was to collect annotations about the similarity of their research goal. The test sets consisted of a list of reference articles and their corresponding top 20 similar articles from PubMed. The task consisted in annotating the similar articles regarding the similarity of their research goal with respect to the one from the corresponding reference article. The assessment of the similarity was based on three labels: "similar", "uncertain", or "not similar". We released two batches of test sets: (a) a first batch of 25 reference articles for five diseases; and (b) a second batch of 80 reference articles for 16 diseases. We collected manual annotations from two teams (RCX and Bf3R) and automatic predictions from two large language models (GPT-4omini and Llama3.3). The preliminary evaluation showed a rather low agreement between the annotators, however, some pairs could potentially be part of a future dataset.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10600 - Biological sciences

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 24th Workshop on Biomedical Language Processing

ISBN

9798891762756

ISSN

—

e-ISSN

—

Počet stran výsledku

8

Strana od-do

388-395

Název nakladatele

Association for Computational Linguistics

Místo vydání

Stroudsburg

Místo konání akce

Vienna, AUSTRIA

Datum konání akce

1. 8. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

001616252100033