Sequence-to-sequence pretraining for a less-resourced Slovenian language

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3AGY8X9D7V" target="_blank" >RIV/00216208:11320/23:GY8X9D7V - isvavai.cz</a>

Result on the web

<a href="https://www.frontiersin.org/articles/10.3389/frai.2023.932519" target="_blank" >https://www.frontiersin.org/articles/10.3389/frai.2023.932519</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.3389/frai.2023.932519" target="_blank" >10.3389/frai.2023.932519</a>

Result language

angličtina

Original language name

Sequence-to-sequence pretraining for a less-resourced Slovenian language

Original language description

"IntroductionLarge pretrained language models have recently conquered the area of natural language processing. As an alternative to predominant masked language modeling introduced in BERT, the T5 model has introduced a more general training objective, namely sequence to sequence transformation, which more naturally fits text generation tasks. The monolingual variants of T5 models have been limited to well-resourced languages, while the massively multilingual T5 model supports 101 languages.MethodsWe trained two different-sized T5-type sequence-to-sequence models for morphologically rich Slovene language with much fewer resources. We analyzed the behavior of new models on 11 tasks, eight classification ones (named entity recognition, sentiment classification, lemmatization, two question answering tasks, two natural language inference tasks, and a coreference resolution task), and three text generation tasks (text simplification and two summarization tasks on different datasets). We compared the new SloT5 models with the multilingual mT5 model, multilingual mBART-50 model, and with four encoder BERT-like models: multilingual BERT, multilingual XLM-RoBERTa, trilingual Croatian-Slovene-English BERT, and monolingual Slovene RoBERTa model.ResultsConcerning the classification tasks, the SloT5 models mostly lag behind the monolingual Slovene SloBERTa model. However, these models are helpful for generative tasks and provide several useful results. In general, the size of models matters, and currently, there is not enough training data for Slovene for successful pretraining of large models.DiscussionWhile the results are obtained on Slovene, we believe that they may generalize to other less-resourced languages, where such models will be built. We make the training and evaluation code, as well as the trained models, publicly available."

Czech name

—

Czech description

—

Type

J_ost - Miscellaneous article in a specialist periodical

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

"Frontiers in Artificial Intelligence"

ISSN

2624-8212

e-ISSN

—

Volume of the periodical

6

Issue of the periodical within the volume

2023

Country of publishing house

US - UNITED STATES

Number of pages

13

Pages from-to

1-13

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-85152679109