The Hidden Space of Transformer Language Adapters
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AI3D3EHG2" target="_blank" >RIV/00216208:11320/25:I3D3EHG2 - isvavai.cz</a>
Result on the web
<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85204448462&partnerID=40&md5=9fb040d4148812570a11cd2297870de0" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85204448462&partnerID=40&md5=9fb040d4148812570a11cd2297870de0</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
The Hidden Space of Transformer Language Adapters
Original language description
We analyze the operation of transformer language adapters, which are small modules trained on top of a frozen language model to adapt its predictions to new target languages. We show that adapted predictions mostly evolve in the source language the model was trained on, while the target language becomes pronounced only in the very last layers of the model. Moreover, the adaptation process is gradual and distributed across layers, where it is possible to skip small groups of adapters without decreasing adaptation performance. Last, we show that adapters operate on top of the model's frozen representation space while largely preserving its structure, rather than on an “isolated” subspace. Our findings provide a deeper view into the adaptation process of language models to new languages, showcasing the constraints imposed on it by the underlying model and introduces practical implications to enhance its efficiency. © 2024 Association for Computational Linguistics.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proc. Annu. Meet. Assoc. Comput Linguist.
ISBN
979-889176094-3
ISSN
0736-587X
e-ISSN
—
Number of pages
20
Pages from-to
6588-6607
Publisher name
Association for Computational Linguistics (ACL)
Place of publication
—
Event location
Hybrid, Bangkok
Event date
Jan 1, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—