Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F22%3AZNLLGNT4" target="_blank" >RIV/00216208:11320/22:ZNLLGNT4 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.hindawi.com/journals/mpe/2022/5483535/" target="_blank" >https://www.hindawi.com/journals/mpe/2022/5483535/</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1155/2022/5483535" target="_blank" >10.1155/2022/5483535</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language

Popis výsledku v původním jazyce

Machine translation makes it easy for people to communicate across languages. Multimodal machine translation is also one of the important directions of research in machine translation, which uses feature information such as images and audio to assist translation models in obtaining higher quality target languages. However, in the vast majority of current research work has been conducted on the basis of commonly used corpora such as English, French, German, less research has been done on low-resource languages, and this has left the translation of low-resource languages relatively behind. This paper selects the English-Hindi and English-Hausa corpus, researched on low-resource language translation. The different models we use for image feature information extraction are fusion of image features with text information in the text encoding process of translation, using image features to provide additional information, and assisting the translation model for translation. Compared with text-only machine translation, the experimental results show that our method improves 3 BLEU in the English-Hindi dataset and improves 0.47 BLEU in the English-Hausa dataset. In addition, we also analyze the effect of image feature information extracted by different feature extraction models on the translation results. Different models pay different attention to each region of the image, and ResNet model is able to extract more feature information compared to VGG model, which is more effective for translation.

Název v anglickém jazyce

Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language

Popis výsledku anglicky

Machine translation makes it easy for people to communicate across languages. Multimodal machine translation is also one of the important directions of research in machine translation, which uses feature information such as images and audio to assist translation models in obtaining higher quality target languages. However, in the vast majority of current research work has been conducted on the basis of commonly used corpora such as English, French, German, less research has been done on low-resource languages, and this has left the translation of low-resource languages relatively behind. This paper selects the English-Hindi and English-Hausa corpus, researched on low-resource language translation. The different models we use for image feature information extraction are fusion of image features with text information in the text encoding process of translation, using image features to provide additional information, and assisting the translation model for translation. Compared with text-only machine translation, the experimental results show that our method improves 3 BLEU in the English-Hindi dataset and improves 0.47 BLEU in the English-Hausa dataset. In addition, we also analyze the effect of image feature information extracted by different feature extraction models on the translation results. Different models pay different attention to each region of the image, and ResNet model is able to extract more feature information compared to VGG model, which is more effective for translation.

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2022

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Mathematical Problems in Engineering

ISSN

1024-123X

e-ISSN

1563-5147

Svazek periodika

2022

Číslo periodika v rámci svazku

2022-8-30

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

9

Strana od-do

1-9

Kód UT WoS článku

000863227700012

EID výsledku v databázi Scopus

2-s2.0-85137866371