Automatic Resource Augmentation for Machine Translation in Low Resource Language: EnIndic Corpus

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3AQ23BRT38" target="_blank" >RIV/00216208:11320/23:Q23BRT38 - isvavai.cz</a>

Result on the web

<a href="https://dl.acm.org/doi/10.1145/3617371" target="_blank" >https://dl.acm.org/doi/10.1145/3617371</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3617371" target="_blank" >10.1145/3617371</a>

Result language

angličtina

Original language name

Automatic Resource Augmentation for Machine Translation in Low Resource Language: EnIndic Corpus

Original language description

"Parallel corpus is the primary ingredient of machine translation. It is required to train the statistical machine translation (SMT) and neural machine translation (NMT) systems. There is a lack of good quality parallel corpus for Hindi to English. Comparable corpora for a given language pair are comparatively easy to find, but this cannot be used directly in SMT or NMT systems. As a result, we generate a parallel corpus from the comparable corpus. For this purpose, the sentences (which are translations of each other) are mined from the comparable corpus to prepare the parallel corpus. The proposed algorithm uses the length of the sentence and word translation model to align sentence pairs that are translations of each other. Then, the sentence pairs that are poor translations of each other (measured by a similarity score based on IBM model 1 translation probability) are filtered out. We apply this algorithm to comparable corpora, which are crawled from speeches of the President and Vice-President of India, and mined parallel corpora out of them. The prepared parallel corpus contains good quality aligned sentences (with 96.338% f-score). Subsequently, incorrect sentence pairs are filtered out manually to make the corpus in qualitative practical use. Finally, we gather various sentences from different sources to prepare the EnIndic corpus, which comprises 1,656,207 English-Hindi sentence pairs (miscellaneous domain). We have deployed this prepared largest English-Hindi parallel corpus at https://github.com/debajyoty/EnIndic.git and the source code at https://github.com/debajyoty/EnIndicSourceCode.git."

Czech name

—

Czech description

—

Type

J_ost - Miscellaneous article in a specialist periodical

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

"ACM Transactions on Asian and Low-Resource Language Information Processing"

ISSN

2375-4699

e-ISSN

—

Volume of the periodical

""

Issue of the periodical within the volume

2023

Country of publishing house

US - UNITED STATES

Number of pages

17

Pages from-to

1-17

UT code for WoS article

—

EID of the result in the Scopus database

—