SI-NLI: A Slovene Natural Language Inference Dataset and its Evaluation

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AZKSAH5GH" target="_blank" >RIV/00216208:11320/25:ZKSAH5GH - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195972966&partnerID=40&md5=099eb30142d64a68f5de66ad0d616240" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195972966&partnerID=40&md5=099eb30142d64a68f5de66ad0d616240</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

SI-NLI: A Slovene Natural Language Inference Dataset and its Evaluation

Original language description

Natural language inference (NLI) is an important language understanding benchmark. Two deficiencies of this benchmark are: i) most existing NLI datasets exist for English and a few other well-resourced languages, and ii) most NLI datasets are formed with a narrow set of annotators' instructions, allowing the prediction models to capture linguistic clues instead of measuring true reasoning capability. We address both issues and introduce SI-NLI, the first dataset for Slovene natural language inference. The dataset is constructed from scratch using knowledgeable annotators with carefully crafted guidelines aiming to avoid commonly encountered problems in existing NLI datasets. We also manually translate the SI-NLI to English to enable cross-lingual model training and evaluation. Using the newly created dataset and its translation, we train and evaluate a variety of large transformer language models in a monolingual and cross-lingual setting. The results indicate that larger models, in general, achieve better performance. The qualitative analysis shows that the SI-NLI dataset is diverse and that there remains plenty of room for improvement even for the largest models. © 2024 ELRA Language Resource Association: CC BY-NC 4.0.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Jt. Int. Conf. Comput. Linguist., Lang. Resour. Eval., LREC-COLING - Main Conf. Proc.

ISBN

978-249381410-4

ISSN

—

e-ISSN

—

Number of pages

12

Pages from-to

14859-14870

Publisher name

European Language Resources Association (ELRA)

Place of publication

—

Event location

Torino, Italia

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—