A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F46747885%3A24220%2F24%3A00012876" target="_blank" >RIV/46747885:24220/24:00012876 - isvavai.cz</a>

Result on the web

<a href="https://asmp-eurasipjournals.springeropen.com/articles/10.1186/s13636-024-00382-2" target="_blank" >https://asmp-eurasipjournals.springeropen.com/articles/10.1186/s13636-024-00382-2</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1186/s13636-024-00382-2" target="_blank" >10.1186/s13636-024-00382-2</a>

Result language

angličtina

Original language name

A lightweight approach to real-time speaker diarization: from audio toward audio-visual data streams

Original language description

This manuscript deals with the task of real-time speaker diarization (SD) for stream-wise data processing. Therefore, in contrast to most of the existing papers, it considers not only the accuracy but also the computational demands of individual investigated methods. We first propose a new lightweight scheme allowing us to perform speaker diarization of streamed audio data. Our approach utilizes a modified residual network with squeeze-and-excitation blocks (SE-ResNet-34) to extract speaker embeddings in an optimized way using cached buffers. These embeddings are subsequently used for voice activity detection (VAD) and block-online k-means clustering with a look-ahead mechanism. The described scheme yields results similar to the reference offline system while operating solely on a CPU with a low real-time factor (RTF) below 0.1 and a constant latency of around 5.5 s. In the next part of the work, our research moves toward much more demanding and complex real-time processing of audio-visual data streams. For this purpose, we extend the above-mentioned scheme for audio data processing by adding an audio-video module. This module utilizes SyncNet combined with visual embeddings for identity tracking. Our resulting multi-modal SD framework then combines the outputs from audio and audio-video modules by using a new overlap-based fusion strategy. It yields diarization error rates that are competitive with the existing state-of-the-art offline audio-visual methods while allowing us to process various audio-video streams, e.g., from Internet or TV broadcasts, in real-time using GPU and with the same latency as for audio data processing.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

20201 - Electrical and electronic engineering

Project

<a href="/en/project/TO01000027" target="_blank" >TO01000027: NORDTRANS - Technology for automatic speech transcription in selected Nordic languages</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Eurasip Journal on Audio, Speech, and Music Processing

ISSN

1687-4722

e-ISSN

—

Volume of the periodical

2024

Issue of the periodical within the volume

1

Country of publishing house

US - UNITED STATES

Number of pages

16

Pages from-to

—

UT code for WoS article

001365828000001

EID of the result in the Scopus database

2-s2.0-85210595217