Improving BERTScore for Machine Translation Evaluation Through Contrastive Learning

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AYD2WDDLX" target="_blank" >RIV/00216208:11320/25:YD2WDDLX - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85194853736&doi=10.1109%2fACCESS.2024.3406993&partnerID=40&md5=57480f39e9117d46d4e0094cf242b313" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85194853736&doi=10.1109%2fACCESS.2024.3406993&partnerID=40&md5=57480f39e9117d46d4e0094cf242b313</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/ACCESS.2024.3406993" target="_blank" >10.1109/ACCESS.2024.3406993</a>

Result language

angličtina

Original language name

Improving BERTScore for Machine Translation Evaluation Through Contrastive Learning

Original language description

BERTScore is an automatic evaluation metric for machine translation. It calculates similarity scores between candidate and reference tokens through embeddings. The quality of embeddings is crucial, but embeddings of low-resource languages tend to be poor. Multilingual pre-trained models can transfer knowledge from rich-resource languages to low-resource languages, but embeddings from these models are not always well aligned. To improve BERTScore for low-resource languages, we attempt to align embeddings by fine-tuning pre-trained models via contrastive learning which shortens the distance between semantically similar sentences and increases the distance between dissimilar sentences. We experiment on Hausa, a low-resource language, in the WMT21 English-Hausa translation task. We conduct fine-tuning on three different pre-trained models (XLM-R, mBERT, LaBSE). Our experimental results show that our proposed method not only achieves higher correlation with human judgments than original BERTScore, but also surpass surface-based metrics such as BLEU, chrF, and the state-of-the-art metric COMET, when fine-tuning mBERT. Moreover, our proposed method generates better embeddings than pre-trained embedding models (E5, BGE, M3E) which are fine-tuned on different NLP tasks. We also extend our experiments to Chinese, a rich-resource language, in an English-Chinese translation task, and further confirms the effectiveness of our method. © 2013 IEEE.

Czech name

—

Czech description

—

Type

J_SC - Article in a specialist periodical, which is included in the SCOPUS database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

IEEE Access

ISSN

2169-3536

e-ISSN

—

Volume of the periodical

12

Issue of the periodical within the volume

2024

Country of publishing house

US - UNITED STATES

Number of pages

11

Pages from-to

77739-77749

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-85194853736