An information theoretic view on selecting linguistic probes

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F20%3A10426991" target="_blank" >RIV/00216208:11320/20:10426991 - isvavai.cz</a>

Result on the web

<a href="https://www.aclweb.org/anthology/2020.emnlp-main.744" target="_blank" >https://www.aclweb.org/anthology/2020.emnlp-main.744</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

An information theoretic view on selecting linguistic probes

Original language description

There is increasing interest in assessing the linguistic knowledge encoded in neural representations. A popular approach is to attach a diagnostic classifier – or “probe” – to perform supervised classification from internal representations. However, how to select a good probe is in debate. Hewitt and Liang (2019) showed that a high performance on diagnostic classification itself is insufficient, because it can be attributed to either “the representation being rich in knowledge”, or “the probe learning the task”, which Pimentel et al. (2020) challenged. We show this dichotomy is valid informationtheoretically. In addition, we find that the methods to construct and select good probes proposed by the two papers, control task (Hewitt and Liang, 2019) and control function (Pimentel et al., 2020), are equivalent – the errors of their approaches are identical (modulo irrelevant terms). Empirically, these two selection criteria lead to results that highly agree with each other.

Czech name

—

Czech description

—

Type

O - Miscellaneous

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2020

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů