Introducing Orthogonal Constraint in Structural Probes

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F21%3A10440558" target="_blank" >RIV/00216208:11320/21:10440558 - isvavai.cz</a>

Result on the web

<a href="https://aclanthology.org/2021.acl-long.36.pdf" target="_blank" >https://aclanthology.org/2021.acl-long.36.pdf</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2021.acl-long.36" target="_blank" >10.18653/v1/2021.acl-long.36</a>

Result language

angličtina

Original language name

Introducing Orthogonal Constraint in Structural Probes

Original language description

With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of word embeddings is performed in order to approximate the topology of dependency structures. In this work, we introduce a new type of structural probing, where the linear projection is decomposed into 1. iso-morphic space rotation; 2. linear scaling that identifies and scales the most relevant dimensions. In addition to syntactic dependency, we evaluate our method on two novel tasks (lexical hypernymy and position in a sentence). We jointly train the probes for multiple tasks and experimentally show that lexical and syntactic information is separated in the representations. Moreover, the orthogonal constraint makes the Structural Probes less vulnerable to memorization.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/GA18-02196S" target="_blank" >GA18-02196S: Linguistic Structure Representation in Neural Networks</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2021

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing

ISBN

978-1-954085-52-7

ISSN

—

e-ISSN

—

Number of pages

15

Pages from-to

428-442

Publisher name

Association for Computational Linguistics

Place of publication

Stroudsburg, PA, USA

Event location

Online

Event date

Aug 2, 2021

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—