Automatic Resource Augmentation for Machine Translation in Low Resource Language: EnIndic Corpus

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3AQ23BRT38" target="_blank" >RIV/00216208:11320/23:Q23BRT38 - isvavai.cz</a>

Výsledek na webu

<a href="https://dl.acm.org/doi/10.1145/3617371" target="_blank" >https://dl.acm.org/doi/10.1145/3617371</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3617371" target="_blank" >10.1145/3617371</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Automatic Resource Augmentation for Machine Translation in Low Resource Language: EnIndic Corpus

Popis výsledku v původním jazyce

"Parallel corpus is the primary ingredient of machine translation. It is required to train the statistical machine translation (SMT) and neural machine translation (NMT) systems. There is a lack of good quality parallel corpus for Hindi to English. Comparable corpora for a given language pair are comparatively easy to find, but this cannot be used directly in SMT or NMT systems. As a result, we generate a parallel corpus from the comparable corpus. For this purpose, the sentences (which are translations of each other) are mined from the comparable corpus to prepare the parallel corpus. The proposed algorithm uses the length of the sentence and word translation model to align sentence pairs that are translations of each other. Then, the sentence pairs that are poor translations of each other (measured by a similarity score based on IBM model 1 translation probability) are filtered out. We apply this algorithm to comparable corpora, which are crawled from speeches of the President and Vice-President of India, and mined parallel corpora out of them. The prepared parallel corpus contains good quality aligned sentences (with 96.338% f-score). Subsequently, incorrect sentence pairs are filtered out manually to make the corpus in qualitative practical use. Finally, we gather various sentences from different sources to prepare the EnIndic corpus, which comprises 1,656,207 English-Hindi sentence pairs (miscellaneous domain). We have deployed this prepared largest English-Hindi parallel corpus at https://github.com/debajyoty/EnIndic.git and the source code at https://github.com/debajyoty/EnIndicSourceCode.git."

Název v anglickém jazyce

Automatic Resource Augmentation for Machine Translation in Low Resource Language: EnIndic Corpus

Popis výsledku anglicky

"Parallel corpus is the primary ingredient of machine translation. It is required to train the statistical machine translation (SMT) and neural machine translation (NMT) systems. There is a lack of good quality parallel corpus for Hindi to English. Comparable corpora for a given language pair are comparatively easy to find, but this cannot be used directly in SMT or NMT systems. As a result, we generate a parallel corpus from the comparable corpus. For this purpose, the sentences (which are translations of each other) are mined from the comparable corpus to prepare the parallel corpus. The proposed algorithm uses the length of the sentence and word translation model to align sentence pairs that are translations of each other. Then, the sentence pairs that are poor translations of each other (measured by a similarity score based on IBM model 1 translation probability) are filtered out. We apply this algorithm to comparable corpora, which are crawled from speeches of the President and Vice-President of India, and mined parallel corpora out of them. The prepared parallel corpus contains good quality aligned sentences (with 96.338% f-score). Subsequently, incorrect sentence pairs are filtered out manually to make the corpus in qualitative practical use. Finally, we gather various sentences from different sources to prepare the EnIndic corpus, which comprises 1,656,207 English-Hindi sentence pairs (miscellaneous domain). We have deployed this prepared largest English-Hindi parallel corpus at https://github.com/debajyoty/EnIndic.git and the source code at https://github.com/debajyoty/EnIndicSourceCode.git."

Druh

J_ost - Ostatní články v recenzovaných periodicích

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

"ACM Transactions on Asian and Low-Resource Language Information Processing"

ISSN

2375-4699

e-ISSN

—

Svazek periodika

""

Číslo periodika v rámci svazku

2023

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

17

Strana od-do

1-17

Kód UT WoS článku

—

EID výsledku v databázi Scopus

—