Investigating UD Treebanks via Dataset Difficulty Measures

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A9PMAMTK2" target="_blank" >RIV/00216208:11320/23:9PMAMTK2 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85159860608&partnerID=40&md5=cbc54358d55d6c2e13939ac8062137cc" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85159860608&partnerID=40&md5=cbc54358d55d6c2e13939ac8062137cc</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Investigating UD Treebanks via Dataset Difficulty Measures

Popis výsledku v původním jazyce

"Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent quality characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, V-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein. © 2023 Association for Computational Linguistics."

Název v anglickém jazyce

Investigating UD Treebanks via Dataset Difficulty Measures

Popis výsledku anglicky

"Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent quality characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, V-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein. © 2023 Association for Computational Linguistics."

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

"EACL - Conf. Eur. Chapter Assoc. Comput. Linguist., Proc. Conf."

ISBN

978-195942944-9

ISSN

—

e-ISSN

—

Počet stran výsledku

14

Strana od-do

1076-1089

Název nakladatele

Association for Computational Linguistics (ACL)

Místo vydání

Dubrovnik

Místo konání akce

Dubrovnik

Datum konání akce

1. 1. 2023

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—