SLTev: Comprehensive Evaluation of Spoken Language Translation
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F21%3A10440518" target="_blank" >RIV/00216208:11320/21:10440518 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2021.eacl-demos.9" target="_blank" >https://aclanthology.org/2021.eacl-demos.9</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.18653/v1/2021.eacl-demos.9" target="_blank" >10.18653/v1/2021.eacl-demos.9</a>
Alternative languages
Result language
angličtina
Original language name
SLTev: Comprehensive Evaluation of Spoken Language Translation
Original language description
Automatic evaluation of Machine Translation(MT) quality has been investigated over several decades. Spoken Language Translation (SLT), especially when simultaneous, needs to consider additional criteria and does not have a standard evaluation procedure and a widely used toolkit. To fill the gap, we introduce SLTev, an open-source tool for assessing SLT in a comprehensive way. SLTev reports the quality, latency, and stability of an SLT candidate output based on the time-stamped transcript and reference translation into a target language. For quality, we rely on sacreBLEU which provides MT evaluation measures such as chrF or BLEU. For latency, we propose two new scoring techniques. For stability, we extend the previously defined measures with a normalized Flicker in our work. We also propose a new averaging of older measures. A preliminary version of SLTev was used in the IWSLT 2020 SHARED TASK. Moreover, a growing collection of test datasets directly accessible by SLTev are provided for system evaluati
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/GX19-26934X" target="_blank" >GX19-26934X: Neural Representations in Multi-modal and Multi-lingual Modeling</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
ISBN
978-1-954085-05-3
ISSN
—
e-ISSN
—
Number of pages
9
Pages from-to
71-79
Publisher name
Association for Computational Linguistics (ACL)
Place of publication
Stroudsburg, PA, USA
Event location
Online
Event date
Apr 19, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—