MultiFlipFormer: A Multimodal Transformer for Emotion Flip Reasoning and Instigator Detection in Therapeutic Conversations

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26220%2F26%3A0200029" target="_blank" >RIV/00216305:26220/26:0200029 - isvavai.cz</a>

Výsledek na webu

<a href="https://ieeexplore.ieee.org/document/11268690/keywords#keywords" target="_blank" >https://ieeexplore.ieee.org/document/11268690/keywords#keywords</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/ICUMT67815.2025.11268690" target="_blank" >10.1109/ICUMT67815.2025.11268690</a>

Jazyk výsledku

angličtina

Název v původním jazyce

MultiFlipFormer: A Multimodal Transformer for Emotion Flip Reasoning and Instigator Detection in Therapeutic Conversations

Popis výsledku v původním jazyce

Understanding the dynamics of emotional change within therapeutic conversations is a key challenge in computational mental health. We present MultiFlipFormer, a multimodal transformer-based architecture designed to model emotion flip reasoning and instigator detection. Unlike prior models that focus on static emotional state classification, MultiFlipFormer captures temporal emotional transitions across dialogue turns and identifies multimodal cues responsible for these shifts. The architecture comprises four core components: (1) emotion transition analysis to model evolving emotional trajectories (e.g., sadness → anger → neutral), (2) multimodal instigator detection that integrates visual cues, textual content, and conversational strategies to uncover emotion flip triggers, (3) cross-modal attention networks to fuse text and visual information, and (4) trajectory prediction to anticipate future emotional states. Evaluated on a large-scale MESC dataset of 23,126 therapeutic samples spanning 7 emotions, 41 flip types, 15 scenarios, and 10 intervention strategies, MultiFlipFormer achieves a final weighted F1-score of 0.828 across tasks. This framework provides a clear and reliable approach that can genuinely support therapy and help in planning better interventions.

Název v anglickém jazyce

MultiFlipFormer: A Multimodal Transformer for Emotion Flip Reasoning and Instigator Detection in Therapeutic Conversations

Popis výsledku anglicky

Understanding the dynamics of emotional change within therapeutic conversations is a key challenge in computational mental health. We present MultiFlipFormer, a multimodal transformer-based architecture designed to model emotion flip reasoning and instigator detection. Unlike prior models that focus on static emotional state classification, MultiFlipFormer captures temporal emotional transitions across dialogue turns and identifies multimodal cues responsible for these shifts. The architecture comprises four core components: (1) emotion transition analysis to model evolving emotional trajectories (e.g., sadness → anger → neutral), (2) multimodal instigator detection that integrates visual cues, textual content, and conversational strategies to uncover emotion flip triggers, (3) cross-modal attention networks to fuse text and visual information, and (4) trajectory prediction to anticipate future emotional states. Evaluated on a large-scale MESC dataset of 23,126 therapeutic samples spanning 7 emotions, 41 flip types, 15 scenarios, and 10 intervention strategies, MultiFlipFormer achieves a final weighted F1-score of 0.828 across tasks. This framework provides a clear and reliable approach that can genuinely support therapy and help in planning better interventions.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

20203 - Telecommunications

Projekt

<a href="/cs/project/VJ02010019" target="_blank" >VJ02010019: Nástroje forenzní expertizy ručně psaného písma</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

2025 17th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT)

ISBN

979-8-3315-7675-2

ISSN

—

e-ISSN

2157-023X

Počet stran výsledku

6

Strana od-do

187-192

Název nakladatele

IEEE

Místo vydání

Florence, Italy

Místo konání akce

Florence, Italy

Datum konání akce

3. 11. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—