Lipreading with LipsID

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F20%3A43959761" target="_blank" >RIV/49777513:23520/20:43959761 - isvavai.cz</a>

Výsledek na webu

<a href="https://link.springer.com/chapter/10.1007%2F978-3-030-60276-5_18" target="_blank" >https://link.springer.com/chapter/10.1007%2F978-3-030-60276-5_18</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-030-60276-5_18" target="_blank" >10.1007/978-3-030-60276-5_18</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Lipreading with LipsID

Popis výsledku v původním jazyce

This paper presents an approach for adaptation of the current visual speech recognition systems. The adaptation technique is based on LipsID features. These features represent a processed area of lips ROI. The features are extracted in a classification task by neural network pre-trained on the dataset-specific to the lip-reading system used for visual speech recognition. The training procedure for LipsID implements ArcFace loss to separate different speakers in the dataset and to provide distinctive features for every one of them. The network uses convolutional layers to extract features from input sequences of speaker images and is designed to take the same input as the lipreading system. Parallel processing of input sequence by LipsID network and lipreading network is followed by a combination of both feature sets and final recognition by Connectionist Temporal Classification (CTC) mechanism. This paper presents results from experiments with the LipNet network by re-implementing the system and comparing it with and without LipsID features. The results show a promising path for future experiments and other systems. The training and testing process of neural networks used in this work utilizes Tensorflow/Keras implementations.

Název v anglickém jazyce

Lipreading with LipsID

Popis výsledku anglicky

This paper presents an approach for adaptation of the current visual speech recognition systems. The adaptation technique is based on LipsID features. These features represent a processed area of lips ROI. The features are extracted in a classification task by neural network pre-trained on the dataset-specific to the lip-reading system used for visual speech recognition. The training procedure for LipsID implements ArcFace loss to separate different speakers in the dataset and to provide distinctive features for every one of them. The network uses convolutional layers to extract features from input sequences of speaker images and is designed to take the same input as the lipreading system. Parallel processing of input sequence by LipsID network and lipreading network is followed by a combination of both feature sets and final recognition by Connectionist Temporal Classification (CTC) mechanism. This paper presents results from experiments with the LipNet network by re-implementing the system and comparing it with and without LipsID features. The results show a promising path for future experiments and other systems. The training and testing process of neural networks used in this work utilizes Tensorflow/Keras implementations.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

20205 - Automation and control systems

Projekt

<a href="/cs/project/LTARF18017" target="_blank" >LTARF18017: AMIR – Multimodální rozhraní založené na gestech a mluvené i znakové řeči pro ovládání asistivního mobilního informačního robota</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2020

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

22nd International Conference, SPECOM 2020, St. Petersburg, Russia, October 7–9, 2020, Proceedings

ISBN

978-3-030-60275-8

ISSN

0302-9743

e-ISSN

1611-3349

Počet stran výsledku

8

Strana od-do

176-183

Název nakladatele

Springer

Místo vydání

Cham

Místo konání akce

St. Petersburg, Russia

Datum konání akce

7. 10. 2020

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—