Script, Language, and Labels: Overcoming Three Discrepancies for Low-Resource Language Specialization

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A4TRGGLZB" target="_blank" >RIV/00216208:11320/23:4TRGGLZB - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85167971966&partnerID=40&md5=6436df972a43f6f9853e3d1b266f7090" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85167971966&partnerID=40&md5=6436df972a43f6f9853e3d1b266f7090</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Script, Language, and Labels: Overcoming Three Discrepancies for Low-Resource Language Specialization

Popis výsledku v původním jazyce

"Although multilingual pretrained models (mPLMs) enabled support of various natural language processing in diverse languages, its limited coverage of 100+ languages lets 6500+ languages remain 'unseen'. One common approach for an unseen language is specializing the model for it as target, by performing additional masked language modeling (MLM) with the target language corpus. However, we argue that, due to the discrepancy from multilingual MLM pretraining, a naïve specialization as such can be suboptimal. Specifically, we pose three discrepancies to overcome. Script and linguistic discrepancy of the target language from the related seen languages, hinder a positive transfer, for which we propose to maximize representation similarity, unlike existing approaches maximizing overlaps. In addition, label space for MLM prediction can vary across languages, for which we propose to reinitialize top layers for a more effective adaptation. Experiments over four different language families and three tasks shows that our method improves the task performance of unseen languages with statistical significance, while previous approach fails to. Copyright © 2023, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved."

Název v anglickém jazyce

Script, Language, and Labels: Overcoming Three Discrepancies for Low-Resource Language Specialization

Popis výsledku anglicky

"Although multilingual pretrained models (mPLMs) enabled support of various natural language processing in diverse languages, its limited coverage of 100+ languages lets 6500+ languages remain 'unseen'. One common approach for an unseen language is specializing the model for it as target, by performing additional masked language modeling (MLM) with the target language corpus. However, we argue that, due to the discrepancy from multilingual MLM pretraining, a naïve specialization as such can be suboptimal. Specifically, we pose three discrepancies to overcome. Script and linguistic discrepancy of the target language from the related seen languages, hinder a positive transfer, for which we propose to maximize representation similarity, unlike existing approaches maximizing overlaps. In addition, label space for MLM prediction can vary across languages, for which we propose to reinitialize top layers for a more effective adaptation. Experiments over four different language families and three tasks shows that our method improves the task performance of unseen languages with statistical significance, while previous approach fails to. Copyright © 2023, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved."

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

"Proc. AAAI Conf. Artif. Intell., AAAI"

ISBN

978-157735880-0

ISSN

—

e-ISSN

—

Počet stran výsledku

10

Strana od-do

13004-13013

Název nakladatele

AAAI Press

Místo vydání

—

Místo konání akce

Melaka, Malaysia

Datum konání akce

1. 1. 2023

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—