Morphosyntactic probing of multilingual BERT models

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3ACPTG9Z2P" target="_blank" >RIV/00216208:11320/23:CPTG9Z2P - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85161321791&doi=10.1017%2fS1351324923000190&partnerID=40&md5=92981f23f267b885a8052ce234546706" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85161321791&doi=10.1017%2fS1351324923000190&partnerID=40&md5=92981f23f267b885a8052ce234546706</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1017/s1351324923000190" target="_blank" >10.1017/s1351324923000190</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Morphosyntactic probing of multilingual BERT models

Popis výsledku v původním jazyce

"We introduce an extensive dataset for multilingual probing of morphological information in language models (247 tasks across 42 languages from 10 families), each consisting of a sentence with a target word and a morphological tag as the desired label, derived from the Universal Dependencies treebanks. We find that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks. We then apply two methods to locate, for each probing task, where the disambiguating information resides in the input. The first is a new perturbation method that masks various parts of context; the second is the classical method of Shapley values. The most intriguing finding that emerges is a strong tendency for the preceding context to hold more information relevant to the prediction than the following context. © The Author(s), 2023. Published by Cambridge University Press."

Název v anglickém jazyce

Morphosyntactic probing of multilingual BERT models

Popis výsledku anglicky

"We introduce an extensive dataset for multilingual probing of morphological information in language models (247 tasks across 42 languages from 10 families), each consisting of a sentence with a target word and a morphological tag as the desired label, derived from the Universal Dependencies treebanks. We find that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks. We then apply two methods to locate, for each probing task, where the disambiguating information resides in the input. The first is a new perturbation method that masks various parts of context; the second is the classical method of Shapley values. The most intriguing finding that emerges is a strong tendency for the preceding context to hold more information relevant to the prediction than the following context. © The Author(s), 2023. Published by Cambridge University Press."

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

"Natural Language Engineering"

ISSN

1351-3249

e-ISSN

—

Svazek periodika

1

Číslo periodika v rámci svazku

1

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

40

Strana od-do

1-40

Kód UT WoS článku

001007784400001

EID výsledku v databázi Scopus

2-s2.0-85161321791