BUT System for the MLC-SLM Challenge

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26230%2F26%3A0199410" target="_blank" >RIV/00216305:26230/26:0199410 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.isca-archive.org/mlcslm_2025/polok25_mlcslm.pdf" target="_blank" >https://www.isca-archive.org/mlcslm_2025/polok25_mlcslm.pdf</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.21437/mlcslm.2025-6" target="_blank" >10.21437/mlcslm.2025-6</a>

Jazyk výsledku

angličtina

Název v původním jazyce

BUT System for the MLC-SLM Challenge

Popis výsledku v původním jazyce

We present a two-speaker automatic speech recognition (ASR) system that combines DiCoW—a diarization-conditioned variant of Whisper—with DiariZen, a diarization pipeline built on top of Pyannote. We first evaluate both systems in out-of-domain (OOD) multilingual scenarios without any fine-tuning. In this scenario, DiariZen consistently outperforms the baseline Pyannote diarization model, demonstrating strong generalization. Despite being fine-tuned on English-only data for target-speaker ASR, DiCoW retains solid multilingual performance,indicating that encoder modifications preserve Whisper’s multilingual capabilities. We then fine-tune both DiCoW and DiariZen on the MLC-SLM challenge data. The fine-tuned DiariZen continues to outperform the fine-tuned Pyannote baseline, while DiCoW sees further gains from domain adaptation. Our final system achieves a micro-average tcpWER/CER of 16.75 % and ranks second in Task 2 of the MLC-SLM challenge. Lastly, we identify several labeling inconsistencies in the training data—such as missing speech segments and incorrect silence annotations—which can hinder diarization fine-tuning. We propose simple mitigation strategies to address these issues and improve system robustness.

Název v anglickém jazyce

BUT System for the MLC-SLM Challenge

Popis výsledku anglicky

We present a two-speaker automatic speech recognition (ASR) system that combines DiCoW—a diarization-conditioned variant of Whisper—with DiariZen, a diarization pipeline built on top of Pyannote. We first evaluate both systems in out-of-domain (OOD) multilingual scenarios without any fine-tuning. In this scenario, DiariZen consistently outperforms the baseline Pyannote diarization model, demonstrating strong generalization. Despite being fine-tuned on English-only data for target-speaker ASR, DiCoW retains solid multilingual performance,indicating that encoder modifications preserve Whisper’s multilingual capabilities. We then fine-tune both DiCoW and DiariZen on the MLC-SLM challenge data. The fine-tuned DiariZen continues to outperform the fine-tuned Pyannote baseline, while DiCoW sees further gains from domain adaptation. Our final system achieves a micro-average tcpWER/CER of 16.75 % and ranks second in Task 2 of the MLC-SLM challenge. Lastly, we identify several labeling inconsistencies in the training data—such as missing speech segments and incorrect silence annotations—which can hinder diarization fine-tuning. We propose simple mitigation strategies to address these issues and improve system robustness.

Druh

O - Ostatní výsledky

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů