A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AS9B5MPZI" target="_blank" >RIV/00216208:11320/25:S9B5MPZI - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85197948717&partnerID=40&md5=59e0699290cda523c714b6054c0cc01f" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85197948717&partnerID=40&md5=59e0699290cda523c714b6054c0cc01f</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets

Original language description

Typologically diverse benchmarks are increasingly created to track the progress achieved in multilingual NLP. Linguistic diversity of these data sets is typically measured as the number of languages or language families included in the sample, but such measures do not consider structural properties of the included languages. In this paper, we propose assessing linguistic diversity of a data set against a reference language sample as a means of maximising linguistic diversity in the long run. We represent languages as sets of features and apply a version of the Jaccard index (Jmm) suitable for comparing sets of measures. In addition to the features extracted from typological data bases, we propose an automatic text-based measure, which can be used as a means of overcoming the well-known problem of data sparsity in manually collected features. Our diversity score is interpretable in terms of linguistic features and can identify the types of languages that are not represented in a data set. Using our method, we analyse a range of popular multilingual data sets (UD, Bible100, mBERT, XTREME, XGLUE, XNLI, XCOPA, TyDiQA, XQuAD). In addition to ranking these data sets, we find, for example, that (poly)synthetic languages are missing in almost all of them. © 2024 Association for Computational Linguistics.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Find. Assoc. Comput. Linguist.: NAACL - Findings

ISBN

979-889176119-3

ISSN

—

e-ISSN

—

Number of pages

16

Pages from-to

3367-3382

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

—

Event location

Mexico City, Mexico

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—