MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A10476212" target="_blank" >RIV/00216208:11320/23:10476212 - isvavai.cz</a>

Result on the web

<a href="https://aclanthology.org/2023.iwslt-1.12.pdf" target="_blank" >https://aclanthology.org/2023.iwslt-1.12.pdf</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

Original language description

There have been several meta-evaluation studies on the correlation between human ratings and offline machine translation (MT) evaluation metrics such as BLEU, chrF2, BertScore and COMET. These metrics have been used to evaluate simultaneous speech translation (SST) but their correlations with human ratings of SST, which has been recently collected as Continuous Ratings (CR), are unclear. In this paper, we leverage the evaluations of candidate systems submitted to the English-German SST task at IWSLT 2022 and conduct an extensive correlation analysis of CR and the aforementioned metrics. Our study reveals that the offline metrics are well correlated with CR and can be reliably used for evaluating machine translation in simultaneous mode, with some limitations on the test set size. We conclude that given the current quality levels of SST, these metrics can be used as proxies for CR, alleviating the need for large scale human evaluation. Additionally, we observe that correlations of the metrics with translation as a reference is significantly higher than with simultaneous interpreting, and thus we recommend the former for reliable evaluation.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

Result was created during the realization of more than one project. More information in the Projects tab.

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the 20th International Conference on Spoken Language Translation

ISBN

978-1-959429-84-5

ISSN

—

e-ISSN

—

Number of pages

11

Pages from-to

169-179

Publisher name

Association for Computational Linguistics

Place of publication

Stroudsburg, USA

Event location

Toronto, Canada

Event date

Jul 13, 2023

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—