Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3ATV6JFSCB" target="_blank" >RIV/00216208:11320/25:TV6JFSCB - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85186095360&doi=10.1109%2fTASLP.2024.3369537&partnerID=40&md5=c3daca4375c1328493ed2c3c8db6a54c" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85186095360&doi=10.1109%2fTASLP.2024.3369537&partnerID=40&md5=c3daca4375c1328493ed2c3c8db6a54c</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/TASLP.2024.3369537" target="_blank" >10.1109/TASLP.2024.3369537</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Popis výsledku v původním jazyce

Neural text-to-speech (TTS) systems have made significant progress in generating natural synthetic speech. However, neural TTS requires large amounts of paired training data, which limits its applicability to a small number of resource-rich languages. Previous work on low-resource TTS has addressed the data hungriness based on transfer learning from a multilingual model to low-resource languages, but it still relies heavily on the availability of paired data for the target languages. In this paper, we propose a text-inductive language adaptation framework for low-resource TTS to address the cost of collecting the paired data for low-resource languages. To inject textual knowledge during transfer learning, our framework employs a two-stage adaptation scheme that utilizes both text-only and supervised data for the target language. In the text-based adaptation stage, we update the language-aware embedding layer with a masked language model objective using text-only data for the target language. In the supervised adaptation stage, the entire TTS model is updated using paired data for the target language. We also propose a graphone-based multilingual training method that jointly uses graphemes and International Phonetic Alphabet symbols (referred to as graphones) for resource-rich languages, while using only graphemes for low-resource languages. This approach facilitates the transfer of pronunciation knowledge from resource-rich to low-resource languages. Through extensive evaluations, we demonstrate that 1) our framework with text-based adaptation outperforms the previous supervised transfer learning approach and 2) the proposed graphone-based training method further improves the performance of both multilingual TTS and low-resource language adaptation. With only 5 minutes of paired data for fine-tuning, our method achieved highly intelligible synthetic speech with the character error rates of around 6 % for a target language. © 2014 IEEE.

Název v anglickém jazyce

Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis

Popis výsledku anglicky

Neural text-to-speech (TTS) systems have made significant progress in generating natural synthetic speech. However, neural TTS requires large amounts of paired training data, which limits its applicability to a small number of resource-rich languages. Previous work on low-resource TTS has addressed the data hungriness based on transfer learning from a multilingual model to low-resource languages, but it still relies heavily on the availability of paired data for the target languages. In this paper, we propose a text-inductive language adaptation framework for low-resource TTS to address the cost of collecting the paired data for low-resource languages. To inject textual knowledge during transfer learning, our framework employs a two-stage adaptation scheme that utilizes both text-only and supervised data for the target language. In the text-based adaptation stage, we update the language-aware embedding layer with a masked language model objective using text-only data for the target language. In the supervised adaptation stage, the entire TTS model is updated using paired data for the target language. We also propose a graphone-based multilingual training method that jointly uses graphemes and International Phonetic Alphabet symbols (referred to as graphones) for resource-rich languages, while using only graphemes for low-resource languages. This approach facilitates the transfer of pronunciation knowledge from resource-rich to low-resource languages. Through extensive evaluations, we demonstrate that 1) our framework with text-based adaptation outperforms the previous supervised transfer learning approach and 2) the proposed graphone-based training method further improves the performance of both multilingual TTS and low-resource language adaptation. With only 5 minutes of paired data for fine-tuning, our method achieved highly intelligible synthetic speech with the character error rates of around 6 % for a target language. © 2014 IEEE.

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

IEEE/ACM Transactions on Audio Speech and Language Processing

ISSN

2329-9290

e-ISSN

—

Svazek periodika

32

Číslo periodika v rámci svazku

2024

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

16

Strana od-do

1829-1844

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-85186095360