Script, Language, and Labels: Overcoming Three Discrepancies for Low-Resource Language Specialization
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A4TRGGLZB" target="_blank" >RIV/00216208:11320/23:4TRGGLZB - isvavai.cz</a>
Result on the web
<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85167971966&partnerID=40&md5=6436df972a43f6f9853e3d1b266f7090" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85167971966&partnerID=40&md5=6436df972a43f6f9853e3d1b266f7090</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
Script, Language, and Labels: Overcoming Three Discrepancies for Low-Resource Language Specialization
Original language description
"Although multilingual pretrained models (mPLMs) enabled support of various natural language processing in diverse languages, its limited coverage of 100+ languages lets 6500+ languages remain 'unseen'. One common approach for an unseen language is specializing the model for it as target, by performing additional masked language modeling (MLM) with the target language corpus. However, we argue that, due to the discrepancy from multilingual MLM pretraining, a naïve specialization as such can be suboptimal. Specifically, we pose three discrepancies to overcome. Script and linguistic discrepancy of the target language from the related seen languages, hinder a positive transfer, for which we propose to maximize representation similarity, unlike existing approaches maximizing overlaps. In addition, label space for MLM prediction can vary across languages, for which we propose to reinitialize top layers for a more effective adaptation. Experiments over four different language families and three tasks shows that our method improves the task performance of unseen languages with statistical significance, while previous approach fails to. Copyright © 2023, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved."
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2023
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
"Proc. AAAI Conf. Artif. Intell., AAAI"
ISBN
978-157735880-0
ISSN
—
e-ISSN
—
Number of pages
10
Pages from-to
13004-13013
Publisher name
AAAI Press
Place of publication
—
Event location
Melaka, Malaysia
Event date
Jan 1, 2023
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—