Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F21%3A10440559" target="_blank" >RIV/00216208:11320/21:10440559 - isvavai.cz</a>

Result on the web

<a href="https://aclanthology.org/2021.emnlp-main.376.pdf" target="_blank" >https://aclanthology.org/2021.emnlp-main.376.pdf</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes

Original language description

State-of-the-art contextual embeddings are obtained from large language models available only for a few languages. For others, we need to learn representations using a multilingual model. There is an ongoing debate on whether multilingual embeddings can be aligned in a space shared across many languages. The novel Orthogonal Structural Probe (Limisiewicz and Mareček, 2021) allows us to answer this question for specific linguistic features and learn a projection based only on mono-lingual annotated datasets. We evaluate syntactic (UD) and lexical (WordNet) structural information encoded inmBERT&apos;s contextual representations for nine diverse languages. We observe that for languages closely related to English, no transformation is needed. The evaluated information is encoded in a shared cross-lingual embedding space. For other languages, it is beneficial to apply orthogonal transformation learned separately for each language. We successfully apply our findings to zero-shot and few-shot cross-lingual parsi

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach<br>I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2021

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)

ISBN

978-1-955917-09-4

ISSN

—

e-ISSN

—

Number of pages

10

Pages from-to

4589-4598

Publisher name

Association for Computational Linguistics

Place of publication

Stroudsburg, PA, USA

Event location

Punta Cana, Dominican Republic

Event date

Nov 7, 2021

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—