Cross-Lingual Transfer with Target Language-Ready Task Adapters

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A9BXK9XME" target="_blank" >RIV/00216208:11320/23:9BXK9XME - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85175471784&partnerID=40&md5=3298fa4bb0804c28b68b479d16cd4a37" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85175471784&partnerID=40&md5=3298fa4bb0804c28b68b479d16cd4a37</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Cross-Lingual Transfer with Target Language-Ready Task Adapters

Popis výsledku v původním jazyce

"Adapters have emerged as a modular and parameter-efficient approach to (zero-shot) cross-lingual transfer. The established MAD-X framework employs separate language and task adapters which can be arbitrarily combined to perform the transfer of any task to any target language. Subsequently, BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X's modularity by creating 'bilingual' adapters specific to the source-target language pair. In this work, we aim to take the best of both worlds by (i) fine-tuning task adapters adapted to the target language(s) (so-called 'target language-ready' (TLR) adapters) to maintain high transfer performance, but (ii) without sacrificing the highly modular design of MAD-X. The main idea of 'target language-ready' adapters is to resolve the training-vs-inference discrepancy of MAD-X: the task adapter 'sees' the target language adapter for the very first time during inference, and thus might not be fully compatible with it. We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters. We evaluate different TLR-based transfer configurations with varying degrees of generality across a suite of standard cross-lingual benchmarks, and find that the most general (and thus most modular) configuration consistently outperforms MAD-X and BAD-X on most tasks and languages. © 2023 Association for Computational Linguistics."

Název v anglickém jazyce

Cross-Lingual Transfer with Target Language-Ready Task Adapters

Popis výsledku anglicky

"Adapters have emerged as a modular and parameter-efficient approach to (zero-shot) cross-lingual transfer. The established MAD-X framework employs separate language and task adapters which can be arbitrarily combined to perform the transfer of any task to any target language. Subsequently, BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X's modularity by creating 'bilingual' adapters specific to the source-target language pair. In this work, we aim to take the best of both worlds by (i) fine-tuning task adapters adapted to the target language(s) (so-called 'target language-ready' (TLR) adapters) to maintain high transfer performance, but (ii) without sacrificing the highly modular design of MAD-X. The main idea of 'target language-ready' adapters is to resolve the training-vs-inference discrepancy of MAD-X: the task adapter 'sees' the target language adapter for the very first time during inference, and thus might not be fully compatible with it. We address this mismatch by exposing the task adapter to the target language adapter during training, and empirically validate several variants of the idea: in the simplest form, we alternate between using the source and target language adapters during task adapter training, which can be generalized to cycling over any set of language adapters. We evaluate different TLR-based transfer configurations with varying degrees of generality across a suite of standard cross-lingual benchmarks, and find that the most general (and thus most modular) configuration consistently outperforms MAD-X and BAD-X on most tasks and languages. © 2023 Association for Computational Linguistics."

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

"Proc. Annu. Meet. Assoc. Comput Linguist."

ISBN

978-195942962-3

ISSN

0736-587X

e-ISSN

—

Počet stran výsledku

18

Strana od-do

176-193

Název nakladatele

Association for Computational Linguistics (ACL)

Místo vydání

—

Místo konání akce

Melaka, Malaysia

Datum konání akce

1. 1. 2023

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—