From scores to insights: Predicting MT errors using reliable metrics and linguistic typology in slavic languages

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216275%3A25410%2F25%3A39923430" target="_blank" >RIV/00216275:25410/25:39923430 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.sciencedirect.com/science/article/pii/S2215016125004571#ack0001" target="_blank" >https://www.sciencedirect.com/science/article/pii/S2215016125004571#ack0001</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.mex.2025.103613" target="_blank" >10.1016/j.mex.2025.103613</a>

Jazyk výsledku

angličtina

Název v původním jazyce

From scores to insights: Predicting MT errors using reliable metrics and linguistic typology in slavic languages

Popis výsledku v původním jazyce

Machine Translation (MT) evaluation plays a crucial role in advancing systems translating into morphologically rich, low-resource languages such as Slovak. Existing automatic evaluation methods typically offer a single quality score, lacking insight into specific error types. A novel linguistically informed methodology that predicts the probability of MT error categories by integrating manual annotation with automatic evaluation metrics is proposed. The method builds on a modified MQM framework adapted for Slovak and employs a dataset of English-to-Slovak translations, combining outputs from statistical and neural MT systems with human reference translations. Manual annotations identified five linguistically motivated error categories. Reli-ability of 68 automatic metrics was assessed using Cronbach&apos;s alpha, correlation coefficients, coefficient of determination (R2), and entropy. Bootstrapped logistic regression models were then developed to predict error occurrence probabilities. The proposed methodology improves the explainability and reliability of automatic MT evaluation by bridging the gap between holistic scoring and detailed error categorization. It significantly reduces the human effort required for quality assessment while maintaining a high degree of linguistic relevance, particularly for complex target languages like Slovak.

Název v anglickém jazyce

From scores to insights: Predicting MT errors using reliable metrics and linguistic typology in slavic languages

Popis výsledku anglicky

Machine Translation (MT) evaluation plays a crucial role in advancing systems translating into morphologically rich, low-resource languages such as Slovak. Existing automatic evaluation methods typically offer a single quality score, lacking insight into specific error types. A novel linguistically informed methodology that predicts the probability of MT error categories by integrating manual annotation with automatic evaluation metrics is proposed. The method builds on a modified MQM framework adapted for Slovak and employs a dataset of English-to-Slovak translations, combining outputs from statistical and neural MT systems with human reference translations. Manual annotations identified five linguistically motivated error categories. Reli-ability of 68 automatic metrics was assessed using Cronbach&apos;s alpha, correlation coefficients, coefficient of determination (R2), and entropy. Bootstrapped logistic regression models were then developed to predict error occurrence probabilities. The proposed methodology improves the explainability and reliability of automatic MT evaluation by bridging the gap between holistic scoring and detailed error categorization. It significantly reduces the human effort required for quality assessment while maintaining a high degree of linguistic relevance, particularly for complex target languages like Slovak.

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

MethodsX

ISSN

—

e-ISSN

2215-0161

Svazek periodika

15

Číslo periodika v rámci svazku

December

Stát vydavatele periodika

NL - Nizozemsko

Počet stran výsledku

9

Strana od-do

103613

Kód UT WoS článku

001584187700002

EID výsledku v databázi Scopus

2-s2.0-105015818486