Sort by Structure: Language Model Ranking as Dependency Probing

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F22%3AQ7PARVER" target="_blank" >RIV/00216208:11320/22:Q7PARVER - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2022.naacl-main.93" target="_blank" >https://aclanthology.org/2022.naacl-main.93</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2022.naacl-main.93" target="_blank" >10.18653/v1/2022.naacl-main.93</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Sort by Structure: Language Model Ranking as Dependency Probing

Popis výsledku v původním jazyce

Making an informed choice of pre-trained language model (LM) is critical for performance, yet environmentally costly, and as such widely underexplored. The field of Computer Vision has begun to tackle encoder ranking, with promising forays into Natural Language Processing, however they lack coverage of linguistic tasks such as structured prediction. We propose probing to rank LMs, specifically for parsing dependencies in a given language, by measuring the degree to which labeled trees are recoverable from an LM's contextualized embeddings. Across 46 typologically and architecturally diverse LM-language pairs, our probing approach predicts the best LM choice 79% of the time using orders of magnitude less compute than training a full parser. Within this study, we identify and analyze one recently proposed decoupled LM—RemBERT—and find it strikingly contains less inherent dependency information, but often yields the best parser after full fine-tuning. Without this outlier our approach identifies the best LM in 89% of cases.

Název v anglickém jazyce

Sort by Structure: Language Model Ranking as Dependency Probing

Popis výsledku anglicky

Making an informed choice of pre-trained language model (LM) is critical for performance, yet environmentally costly, and as such widely underexplored. The field of Computer Vision has begun to tackle encoder ranking, with promising forays into Natural Language Processing, however they lack coverage of linguistic tasks such as structured prediction. We propose probing to rank LMs, specifically for parsing dependencies in a given language, by measuring the degree to which labeled trees are recoverable from an LM's contextualized embeddings. Across 46 typologically and architecturally diverse LM-language pairs, our probing approach predicts the best LM choice 79% of the time using orders of magnitude less compute than training a full parser. Within this study, we identify and analyze one recently proposed decoupled LM—RemBERT—and find it strikingly contains less inherent dependency information, but often yields the best parser after full fine-tuning. Without this outlier our approach identifies the best LM in 89% of cases.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2022

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

ISBN

978-1-955917-74-2

ISSN

—

e-ISSN

—

Počet stran výsledku

12

Strana od-do

1296-1307

Název nakladatele

Association for Computational Linguistics

Místo vydání

—

Místo konání akce

Seattle, United States

Datum konání akce

1. 1. 2022

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—