Quiz-Based Evaluation of Machine Translation

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F11%3A10107804" target="_blank" >RIV/00216208:11320/11:10107804 - isvavai.cz</a>
Result on the web
<a href="http://dx.doi.org/10.2478/v10108-011-0006-1" target="_blank" >http://dx.doi.org/10.2478/v10108-011-0006-1</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.2478/v10108-011-0006-1" target="_blank" >10.2478/v10108-011-0006-1</a>

Result language
angličtina
Original language name
Quiz-Based Evaluation of Machine Translation
Original language description
This paper proposes a new method of manual evaluation for statistical machine translation, the so-called quiz-based evaluation, estimating whether people are able to extract information from machine-translated texts reliably. We apply the method to two commercial and two experimental MT systems that participated in WMT 2010 in English-to-Czech translation. We report inter-annotator agreement for the evaluation as well as the outcomes of the individual systems. The quiz-based evaluation suggests rather different ranking of the systems compared to the WMT 2010 manual and automatic metrics. We also see that overall, MT quality is becoming acceptable for obtaining information from the text: about 80% of questions can be answered correctly given only machine-translated text.
Czech name
—
Czech description
—

Type
J<sub>x</sub> - Unclassified - Peer-reviewed scientific article (Jimp, Jsc and Jost)
CEP classification
AI - Linguistics
OECD FORD branch
—

Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year
2011
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Similar results(10)