Measuring Memorization Effect in Word-Level Neural Networks Probing

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F20%3A10424498" target="_blank" >RIV/00216208:11320/20:10424498 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1007/978-3-030-58323-1_19" target="_blank" >https://doi.org/10.1007/978-3-030-58323-1_19</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-030-58323-1_19" target="_blank" >10.1007/978-3-030-58323-1_19</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Measuring Memorization Effect in Word-Level Neural Networks Probing

Popis výsledku v původním jazyce

Multiple studies have probed representations emerging in neural networks trained for end-to-end NLP tasks and examined what word-level linguistic information may be encoded in the representations. In classical probing, a classifier is trained on the representations to extract the target linguistic information. However, there is a threat of the classifier simply memorizing the linguistic labels for individual words, instead of extracting the linguistic abstractions from the representations, thus reporting false positive results. While considerable efforts have been made to minimize the memorization problem, the task of actually measuring the amount of memorization happening in the classifier has been understudied so far. In our work, we propose a simple general method for measuring the memorization effect, based on a symmetric selection of comparable sets of test words seen versus unseen in training. Our method can be used to explicitly quantify the amount of memorization happening in a probing setup,

Název v anglickém jazyce

Measuring Memorization Effect in Word-Level Neural Networks Probing

Popis výsledku anglicky

Multiple studies have probed representations emerging in neural networks trained for end-to-end NLP tasks and examined what word-level linguistic information may be encoded in the representations. In classical probing, a classifier is trained on the representations to extract the target linguistic information. However, there is a threat of the classifier simply memorizing the linguistic labels for individual words, instead of extracting the linguistic abstractions from the representations, thus reporting false positive results. While considerable efforts have been made to minimize the memorization problem, the task of actually measuring the amount of memorization happening in the classifier has been understudied so far. In our work, we propose a simple general method for measuring the memorization effect, based on a symmetric selection of comparable sets of test words seen versus unseen in training. Our method can be used to explicitly quantify the amount of memorization happening in a probing setup,

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GA18-02196S" target="_blank" >GA18-02196S: Reprezentace lingvistické struktury v neuronových sítích</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2020

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

23rd International Conference on Text, Speech and Dialogue

ISBN

978-3-030-58322-4

ISSN

0302-9743

e-ISSN

—

Počet stran výsledku

9

Strana od-do

180-188

Název nakladatele

Springer

Místo vydání

Cham, Switzerland

Místo konání akce

Brno, Czechia

Datum konání akce

8. 9. 2020

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—