From scores to insights: Predicting MT errors using reliable metrics and linguistic typology in slavic languages

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216275%3A25410%2F25%3A39923430" target="_blank" >RIV/00216275:25410/25:39923430 - isvavai.cz</a>

Result on the web

<a href="https://www.sciencedirect.com/science/article/pii/S2215016125004571#ack0001" target="_blank" >https://www.sciencedirect.com/science/article/pii/S2215016125004571#ack0001</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.mex.2025.103613" target="_blank" >10.1016/j.mex.2025.103613</a>

Result language

angličtina

Original language name

From scores to insights: Predicting MT errors using reliable metrics and linguistic typology in slavic languages

Original language description

Machine Translation (MT) evaluation plays a crucial role in advancing systems translating into morphologically rich, low-resource languages such as Slovak. Existing automatic evaluation methods typically offer a single quality score, lacking insight into specific error types. A novel linguistically informed methodology that predicts the probability of MT error categories by integrating manual annotation with automatic evaluation metrics is proposed. The method builds on a modified MQM framework adapted for Slovak and employs a dataset of English-to-Slovak translations, combining outputs from statistical and neural MT systems with human reference translations. Manual annotations identified five linguistically motivated error categories. Reli-ability of 68 automatic metrics was assessed using Cronbach&apos;s alpha, correlation coefficients, coefficient of determination (R2), and entropy. Bootstrapped logistic regression models were then developed to predict error occurrence probabilities. The proposed methodology improves the explainability and reliability of automatic MT evaluation by bridging the gap between holistic scoring and detailed error categorization. It significantly reduces the human effort required for quality assessment while maintaining a high degree of linguistic relevance, particularly for complex target languages like Slovak.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

MethodsX

ISSN

—

e-ISSN

2215-0161

Volume of the periodical

15

Issue of the periodical within the volume

December

Country of publishing house

NL - THE KINGDOM OF THE NETHERLANDS

Number of pages

9

Pages from-to

103613

UT code for WoS article

001584187700002

EID of the result in the Scopus database

2-s2.0-105015818486