Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3AE4LEH4U7" target="_blank" >RIV/00216208:11320/23:E4LEH4U7 - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2022.naacl-main.255" target="_blank" >https://aclanthology.org/2022.naacl-main.255</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2022.naacl-main.255" target="_blank" >10.18653/v1/2022.naacl-main.255</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Popis výsledku v původním jazyce

"Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-Mod) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages."

Název v anglickém jazyce

Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Popis výsledku anglicky

"Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this issue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language-specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X-Mod) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also enables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages."

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"

ISBN

978-1-955917-71-1

ISSN

—

e-ISSN

—

Počet stran výsledku

17

Strana od-do

3479-3495

Název nakladatele

arXiv

Místo vydání

Seattle, USA

Místo konání akce

Seattle, USA

Datum konání akce

1. 1. 2023

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—