On Evaluation of Natural Language Processing Tasks: Is Gold Standard Evaluation Methodology a Good Solution?

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14330%2F16%3A00087757" target="_blank" >RIV/00216224:14330/16:00087757 - isvavai.cz</a>
Result on the web
—
DOI - Digital Object Identifier
—

Result language
angličtina
Original language name
On Evaluation of Natural Language Processing Tasks: Is Gold Standard Evaluation Methodology a Good Solution?
Original language description
The paper discusses problems in state of the art evaluation methods used in natural language processing (NLP). Usually, some form of gold standard data is used for evaluation of various NLP tasks, ranging from morphological annotation to semantic analysis. We discuss problems and validity of this type of evaluation, for various tasks, and illustrate the problems on examples. Then we propose using application-driven evaluations, wherever it is possible. Although it is more expensive, more complicated and not so precise, it is the only way to find out if a particular tool is useful at all.
Czech name
—
Czech description
—

Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year
2016
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection
Proceedings of the 8th International Conference on Agents and Artificial Intelligence
ISBN
9789897581724
ISSN
—
e-ISSN
—
Number of pages
6
Pages from-to
540-545
Publisher name
SCITEPRESS
Place of publication
Rome
Event location
Rome
Event date
Jan 1, 2016
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—

Similar results(10)