Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F22%3AIFI2Y9FY" target="_blank" >RIV/00216208:11320/22:IFI2Y9FY - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1162/tacl_a_00467" target="_blank" >https://doi.org/10.1162/tacl_a_00467</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1162/tacl_a_00467" target="_blank" >10.1162/tacl_a_00467</a>

Result language

angličtina

Original language name

Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation

Original language description

Common designs of model evaluation typically focus on monolingual settings, where different models are compared according to their performance on a single data set that is assumed to be representative of all possible data for the task at hand. While this may be reasonable for a large data set, this assumption is difficult to maintain in low-resource scenarios, where artifacts of the data collection can yield data sets that are outliers, potentially making conclusions about model performance coincidental. To address these concerns, we investigate model generalizability in crosslinguistic low-resource scenarios. Using morphological segmentation as the test case, we compare three broad classes of models with different parameterizations, taking data from 11 languages across 6 language families. In each experimental setting, we evaluate all models on a first data set, then examine their performance consistency when introducing new randomly sampled data sets with the same size and when applying the trained models to unseen test sets of varying sizes. The results demonstrate that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data set size. Among the characteristics that we studied, the ratio of morpheme overlap and that of the average number of morphemes per word between the training and test sets are the two most prominent factors. Our findings suggest that future work should adopt random sampling to construct data sets with different sizes in order to make more responsible claims about model evaluation.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2022

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Transactions of the Association for Computational Linguistics [online]

ISSN

2307-387X

e-ISSN

1988-2971

Volume of the periodical

10

Issue of the periodical within the volume

2022-4-6

Country of publishing house

US - UNITED STATES

Number of pages

21

Pages from-to

393-413

UT code for WoS article

000923414000004

EID of the result in the Scopus database

2-s2.0-85128885946