Probing the Emergence of Cross-lingual Alignment during LLM Training

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3ADUBYWBKX" target="_blank" >RIV/00216208:11320/25:DUBYWBKX - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85205292172&partnerID=40&md5=ad80761ce6b51e896ec97a724df353c6" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85205292172&partnerID=40&md5=ad80761ce6b51e896ec97a724df353c6</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Probing the Emergence of Cross-lingual Alignment during LLM Training

Original language description

Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. We speculate that this is predicated on their ability to align languages without explicit supervision from parallel sentences. While representations of translationally equivalent sentences in different languages are known to be similar after convergence, however, it remains unclear how such cross-lingual alignment emerges during pretraining of LLMs. Our study leverages intrinsic probing techniques, which identify which subsets of neurons encode linguistic features, to correlate the degree of cross-lingual neuron overlap with the zero-shot cross-lingual transfer performance for a given model. In particular, we rely on checkpoints of BLOOM, a multilingual autoregressive LLM, across different training steps and model scales. We observe a high correlation between neuron overlap and downstream performance, which supports our hypothesis on the conditions leading to effective cross-lingual transfer. Interestingly, we also detect a degradation of both implicit alignment and multilingual abilities in certain phases of the pre-training process, providing new insights into the multilingual pretraining dynamics. © 2024 Association for Computational Linguistics.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proc. Annu. Meet. Assoc. Comput Linguist.

ISBN

979-889176099-8

ISSN

0736-587X

e-ISSN

—

Number of pages

15

Pages from-to

12159-12173

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

—

Event location

Hybrid, Bangkok

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—