TRANSMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AIK7NAQMU" target="_blank" >RIV/00216208:11320/26:IK7NAQMU - isvavai.cz</a>

Výsledek na webu

—

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

TRANSMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data

Popis výsledku v původním jazyce

Transliterating related languages that use different scripts into a common script is effective for improving crosslingual transfer in downstream tasks. However, this methodology often makes pretraining a model from scratch unavoidable, as transliteration brings about new subwords not covered in existing multilingual pretrained language models (mPLMs). This is undesirable because it requires a large computation budget. A more promising way is to make full use of available mPLMs. To this end, this paper proposes a simple but effective framework: Transliterate-Merge-Initialize (TRANSMI). TRANSMI can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. TRANSMI has three stages: (a) transliterate the vocabulary of an mPLM into a common script; (b) merge the new vocabulary with the original vocabulary; and (c) initialize the embeddings of the new subwords. We apply TRANSMI to three strong recent mPLMs. Our experiments demonstrate that TRANSMI not only preserves the mPLM's ability to handle non-transliterated data, but also enables it to effectively process transliterated data, thereby facilitating crosslingual transfer across scripts. The results show consistent improvements of 3% to 34% for different mPLMs and tasks. We make our code and models publicly available at https://github.com/cisnlp/TransMI. © 2025 Association for Computational Linguistics.

Název v anglickém jazyce

TRANSMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data

Popis výsledku anglicky

Transliterating related languages that use different scripts into a common script is effective for improving crosslingual transfer in downstream tasks. However, this methodology often makes pretraining a model from scratch unavoidable, as transliteration brings about new subwords not covered in existing multilingual pretrained language models (mPLMs). This is undesirable because it requires a large computation budget. A more promising way is to make full use of available mPLMs. To this end, this paper proposes a simple but effective framework: Transliterate-Merge-Initialize (TRANSMI). TRANSMI can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. TRANSMI has three stages: (a) transliterate the vocabulary of an mPLM into a common script; (b) merge the new vocabulary with the original vocabulary; and (c) initialize the embeddings of the new subwords. We apply TRANSMI to three strong recent mPLMs. Our experiments demonstrate that TRANSMI not only preserves the mPLM's ability to handle non-transliterated data, but also enables it to effectively process transliterated data, thereby facilitating crosslingual transfer across scripts. The results show consistent improvements of 3% to 34% for different mPLMs and tasks. We make our code and models publicly available at https://github.com/cisnlp/TransMI. © 2025 Association for Computational Linguistics.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proc. Main Conf. Int. Conf. Comput. Linguist., COLING

ISBN

979-8-89176-196-4

ISSN

—

e-ISSN

—

Počet stran výsledku

27

Strana od-do

469-495

Název nakladatele

Association for Computational Linguistics (ACL)

Místo vydání

—

Místo konání akce

Abu Dhabi

Datum konání akce

1. 1. 2026

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—