Investigating UD Treebanks via Dataset Difficulty Measures

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A9PMAMTK2" target="_blank" >RIV/00216208:11320/23:9PMAMTK2 - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85159860608&partnerID=40&md5=cbc54358d55d6c2e13939ac8062137cc" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85159860608&partnerID=40&md5=cbc54358d55d6c2e13939ac8062137cc</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Investigating UD Treebanks via Dataset Difficulty Measures

Original language description

"Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent quality characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, V-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein. © 2023 Association for Computational Linguistics."

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

"EACL - Conf. Eur. Chapter Assoc. Comput. Linguist., Proc. Conf."

ISBN

978-195942944-9

ISSN

—

e-ISSN

—

Number of pages

14

Pages from-to

1076-1089

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

Dubrovnik

Event location

Dubrovnik

Event date

Jan 1, 2023

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—