Introducing Orthogonal Constraint in Structural Probes

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F21%3A10440558" target="_blank" >RIV/00216208:11320/21:10440558 - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2021.acl-long.36.pdf" target="_blank" >https://aclanthology.org/2021.acl-long.36.pdf</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2021.acl-long.36" target="_blank" >10.18653/v1/2021.acl-long.36</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Introducing Orthogonal Constraint in Structural Probes

Popis výsledku v původním jazyce

With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of word embeddings is performed in order to approximate the topology of dependency structures. In this work, we introduce a new type of structural probing, where the linear projection is decomposed into 1. iso-morphic space rotation; 2. linear scaling that identifies and scales the most relevant dimensions. In addition to syntactic dependency, we evaluate our method on two novel tasks (lexical hypernymy and position in a sentence). We jointly train the probes for multiple tasks and experimentally show that lexical and syntactic information is separated in the representations. Moreover, the orthogonal constraint makes the Structural Probes less vulnerable to memorization.

Název v anglickém jazyce

Introducing Orthogonal Constraint in Structural Probes

Popis výsledku anglicky

With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of word embeddings is performed in order to approximate the topology of dependency structures. In this work, we introduce a new type of structural probing, where the linear projection is decomposed into 1. iso-morphic space rotation; 2. linear scaling that identifies and scales the most relevant dimensions. In addition to syntactic dependency, we evaluate our method on two novel tasks (lexical hypernymy and position in a sentence). We jointly train the probes for multiple tasks and experimentally show that lexical and syntactic information is separated in the representations. Moreover, the orthogonal constraint makes the Structural Probes less vulnerable to memorization.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GA18-02196S" target="_blank" >GA18-02196S: Reprezentace lingvistické struktury v neuronových sítích</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

ISBN

978-1-954085-52-7

ISSN

—

e-ISSN

—

Počet stran výsledku

15

Strana od-do

428-442

Název nakladatele

Association for Computational Linguistics

Místo vydání

Stroudsburg, PA, USA

Místo konání akce

Online

Datum konání akce

2. 8. 2021

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—