Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3ATV6JFSCB" target="_blank" >RIV/00216208:11320/25:TV6JFSCB - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85186095360&doi=10.1109%2fTASLP.2024.3369537&partnerID=40&md5=c3daca4375c1328493ed2c3c8db6a54c" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85186095360&doi=10.1109%2fTASLP.2024.3369537&partnerID=40&md5=c3daca4375c1328493ed2c3c8db6a54c</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/TASLP.2024.3369537" target="_blank" >10.1109/TASLP.2024.3369537</a>

Result language

angličtina

Original language name

Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Original language description

Neural text-to-speech (TTS) systems have made significant progress in generating natural synthetic speech. However, neural TTS requires large amounts of paired training data, which limits its applicability to a small number of resource-rich languages. Previous work on low-resource TTS has addressed the data hungriness based on transfer learning from a multilingual model to low-resource languages, but it still relies heavily on the availability of paired data for the target languages. In this paper, we propose a text-inductive language adaptation framework for low-resource TTS to address the cost of collecting the paired data for low-resource languages. To inject textual knowledge during transfer learning, our framework employs a two-stage adaptation scheme that utilizes both text-only and supervised data for the target language. In the text-based adaptation stage, we update the language-aware embedding layer with a masked language model objective using text-only data for the target language. In the supervised adaptation stage, the entire TTS model is updated using paired data for the target language. We also propose a graphone-based multilingual training method that jointly uses graphemes and International Phonetic Alphabet symbols (referred to as graphones) for resource-rich languages, while using only graphemes for low-resource languages. This approach facilitates the transfer of pronunciation knowledge from resource-rich to low-resource languages. Through extensive evaluations, we demonstrate that 1) our framework with text-based adaptation outperforms the previous supervised transfer learning approach and 2) the proposed graphone-based training method further improves the performance of both multilingual TTS and low-resource language adaptation. With only 5 minutes of paired data for fine-tuning, our method achieved highly intelligible synthetic speech with the character error rates of around 6 % for a target language. © 2014 IEEE.

Czech name

—

Czech description

—

Type

J_SC - Article in a specialist periodical, which is included in the SCOPUS database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

IEEE/ACM Transactions on Audio Speech and Language Processing

ISSN

2329-9290

e-ISSN

—

Volume of the periodical

32

Issue of the periodical within the volume

2024

Country of publishing house

US - UNITED STATES

Number of pages

16

Pages from-to

1829-1844

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-85186095360