Improving BERTScore for Machine Translation Evaluation Through Contrastive Learning

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AYD2WDDLX" target="_blank" >RIV/00216208:11320/25:YD2WDDLX - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85194853736&doi=10.1109%2fACCESS.2024.3406993&partnerID=40&md5=57480f39e9117d46d4e0094cf242b313" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85194853736&doi=10.1109%2fACCESS.2024.3406993&partnerID=40&md5=57480f39e9117d46d4e0094cf242b313</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/ACCESS.2024.3406993" target="_blank" >10.1109/ACCESS.2024.3406993</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Improving BERTScore for Machine Translation Evaluation Through Contrastive Learning

Popis výsledku v původním jazyce

BERTScore is an automatic evaluation metric for machine translation. It calculates similarity scores between candidate and reference tokens through embeddings. The quality of embeddings is crucial, but embeddings of low-resource languages tend to be poor. Multilingual pre-trained models can transfer knowledge from rich-resource languages to low-resource languages, but embeddings from these models are not always well aligned. To improve BERTScore for low-resource languages, we attempt to align embeddings by fine-tuning pre-trained models via contrastive learning which shortens the distance between semantically similar sentences and increases the distance between dissimilar sentences. We experiment on Hausa, a low-resource language, in the WMT21 English-Hausa translation task. We conduct fine-tuning on three different pre-trained models (XLM-R, mBERT, LaBSE). Our experimental results show that our proposed method not only achieves higher correlation with human judgments than original BERTScore, but also surpass surface-based metrics such as BLEU, chrF, and the state-of-the-art metric COMET, when fine-tuning mBERT. Moreover, our proposed method generates better embeddings than pre-trained embedding models (E5, BGE, M3E) which are fine-tuned on different NLP tasks. We also extend our experiments to Chinese, a rich-resource language, in an English-Chinese translation task, and further confirms the effectiveness of our method. © 2013 IEEE.

Název v anglickém jazyce

Improving BERTScore for Machine Translation Evaluation Through Contrastive Learning

Popis výsledku anglicky

BERTScore is an automatic evaluation metric for machine translation. It calculates similarity scores between candidate and reference tokens through embeddings. The quality of embeddings is crucial, but embeddings of low-resource languages tend to be poor. Multilingual pre-trained models can transfer knowledge from rich-resource languages to low-resource languages, but embeddings from these models are not always well aligned. To improve BERTScore for low-resource languages, we attempt to align embeddings by fine-tuning pre-trained models via contrastive learning which shortens the distance between semantically similar sentences and increases the distance between dissimilar sentences. We experiment on Hausa, a low-resource language, in the WMT21 English-Hausa translation task. We conduct fine-tuning on three different pre-trained models (XLM-R, mBERT, LaBSE). Our experimental results show that our proposed method not only achieves higher correlation with human judgments than original BERTScore, but also surpass surface-based metrics such as BLEU, chrF, and the state-of-the-art metric COMET, when fine-tuning mBERT. Moreover, our proposed method generates better embeddings than pre-trained embedding models (E5, BGE, M3E) which are fine-tuned on different NLP tasks. We also extend our experiments to Chinese, a rich-resource language, in an English-Chinese translation task, and further confirms the effectiveness of our method. © 2013 IEEE.

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

IEEE Access

ISSN

2169-3536

e-ISSN

—

Svazek periodika

12

Číslo periodika v rámci svazku

2024

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

11

Strana od-do

77739-77749

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-85194853736