Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F22%3AR54RBYBP" target="_blank" >RIV/00216208:11320/22:R54RBYBP - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2022.naacl-main.114" target="_blank" >https://aclanthology.org/2022.naacl-main.114</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.18653/v1/2022.naacl-main.114" target="_blank" >10.18653/v1/2022.naacl-main.114</a>
Alternative languages
Result language
angličtina
Original language name
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models
Original language description
The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. However, it remains unclear how these models learn to generalise across languages. In this work, we conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar. In particular, we investigate whether morphosyntactic information is encoded in the same subset of neurons in different languages.We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe. Our findings show that the cross-lingual overlap between neurons is significant, but its extent may vary across categories and depends on language proximity and pre-training data size.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2022
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
ISBN
978-1-955917-71-1
ISSN
—
e-ISSN
—
Number of pages
10
Pages from-to
1589-1598
Publisher name
Association for Computational Linguistics
Place of publication
—
Event location
Seattle, United States
Event date
Jan 1, 2022
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—