Naturalistic Causal Probing for Morpho-Syntax

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3AJ3XGHLXI" target="_blank" >RIV/00216208:11320/23:J3XGHLXI - isvavai.cz</a>

Výsledek na webu

<a href="https://direct.mit.edu/tacl/article-abstract/doi/10.1162/tacl_a_00554/115895" target="_blank" >https://direct.mit.edu/tacl/article-abstract/doi/10.1162/tacl_a_00554/115895</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1162/tacl_a_00554" target="_blank" >10.1162/tacl_a_00554</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Naturalistic Causal Probing for Morpho-Syntax

Popis výsledku v původním jazyce

"Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing. However, there is still a lack of understanding of the limitations and weaknesses of various types of probes. In this work, we suggest a strategy for input-level intervention on naturalistic sentences. Using our approach, we intervene on the morpho-syntactic features of a sentence, while keeping the rest of the sentence unchanged. Such an intervention allows us to causally probe pre-trained models. We apply our naturalistic causal probing framework to analyze the effects of grammatical gender and number on contextualized representations extracted from three pre-trained models in Spanish, the multilingual versions of BERT, RoBERTa, and GPT-2. Our experiments suggest that naturalistic interventions lead to stable estimates of the causal effects of various linguistic properties. Moreover, our experiments demonstrate the importance of naturalistic causal probing when analyzing pre-trained models."

Název v anglickém jazyce

Naturalistic Causal Probing for Morpho-Syntax

Popis výsledku anglicky

"Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing. However, there is still a lack of understanding of the limitations and weaknesses of various types of probes. In this work, we suggest a strategy for input-level intervention on naturalistic sentences. Using our approach, we intervene on the morpho-syntactic features of a sentence, while keeping the rest of the sentence unchanged. Such an intervention allows us to causally probe pre-trained models. We apply our naturalistic causal probing framework to analyze the effects of grammatical gender and number on contextualized representations extracted from three pre-trained models in Spanish, the multilingual versions of BERT, RoBERTa, and GPT-2. Our experiments suggest that naturalistic interventions lead to stable estimates of the causal effects of various linguistic properties. Moreover, our experiments demonstrate the importance of naturalistic causal probing when analyzing pre-trained models."

Druh

J_ost - Ostatní články v recenzovaných periodicích

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

"Transactions of the Association for Computational Linguistics"

ISSN

2307-387X

e-ISSN

—

Svazek periodika

11

Číslo periodika v rámci svazku

2023

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

20

Strana od-do

384-403

Kód UT WoS článku

—

EID výsledku v databázi Scopus

—