Multimodal speech recognition: increasing accuracy using high speed video data

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F18%3A43952641" target="_blank" >RIV/49777513:23520/18:43952641 - isvavai.cz</a>

Result on the web

<a href="http://dx.doi.org/10.1007/s12193-018-0267-1" target="_blank" >http://dx.doi.org/10.1007/s12193-018-0267-1</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s12193-018-0267-1" target="_blank" >10.1007/s12193-018-0267-1</a>

Result language

angličtina

Original language name

Multimodal speech recognition: increasing accuracy using high speed video data

Original language description

To date, multimodal speech recognition systems based on the processing of audio and video signals show significantly better results than their unimodal counterparts. In general, researchers divide the solution of the audio–visual speech recognition problem into two parts. First, in extracting the most informative features from each modality and second, in the most successful way of fusion both modalities. Ultimately, this leads to an improvement in the accuracy of speech recognition. Almost all modern studies use this approach with video data of a standard recording speed of 25 frames per second. The choice of such a recording speed is easily explained, since the vast majority of existing audio–visual databases are recorded with this rate. However, it should be noticed that the number of 25 frames per second is a world standard for many areas and has never been specifically calculated for speech recognition tasks. The main purpose of this study is to investigate the effect brought by the high-speed video data (up to 200 frames per second) on the speech recognition accuracy. And also to find out whether the use of a high-speed video camera makes the speech recognition systems more robust to acoustical noise. To this end, we recorded a database of audio–visual Russian speech with high-speed video recordings, which consists of records of 20 speakers, each of them pronouncing 200 phrases of continuous Russian speech. Experiments performed on this database showed an improvement in the absolute speech recognition rate up to 3.10%. We also proved that the use of the high-speed camera with 200 fps allows achieving better recognition results under different acoustically noisy conditions (signal-to-noise ratio varied between 40 and 0 dB) with different types of noise (e.g. white noise, babble noise).

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

20205 - Automation and control systems

Project

<a href="/en/project/LO1506" target="_blank" >LO1506: Sustainability support of the centre NTIS - New Technologies for the Information Society</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2018

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Journal on Multimodal User Interfaces

ISSN

1783-7677

e-ISSN

—

Volume of the periodical

12

Issue of the periodical within the volume

4

Country of publishing house

US - UNITED STATES

Number of pages

10

Pages from-to

319-328

UT code for WoS article

000448519400006

EID of the result in the Scopus database

2-s2.0-85051679221