MultiFlipFormer: A Multimodal Transformer for Emotion Flip Reasoning and Instigator Detection in Therapeutic Conversations

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26220%2F26%3A0200029" target="_blank" >RIV/00216305:26220/26:0200029 - isvavai.cz</a>

Result on the web

<a href="https://ieeexplore.ieee.org/document/11268690/keywords#keywords" target="_blank" >https://ieeexplore.ieee.org/document/11268690/keywords#keywords</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/ICUMT67815.2025.11268690" target="_blank" >10.1109/ICUMT67815.2025.11268690</a>

Result language

angličtina

Original language name

MultiFlipFormer: A Multimodal Transformer for Emotion Flip Reasoning and Instigator Detection in Therapeutic Conversations

Original language description

Understanding the dynamics of emotional change within therapeutic conversations is a key challenge in computational mental health. We present MultiFlipFormer, a multimodal transformer-based architecture designed to model emotion flip reasoning and instigator detection. Unlike prior models that focus on static emotional state classification, MultiFlipFormer captures temporal emotional transitions across dialogue turns and identifies multimodal cues responsible for these shifts. The architecture comprises four core components: (1) emotion transition analysis to model evolving emotional trajectories (e.g., sadness → anger → neutral), (2) multimodal instigator detection that integrates visual cues, textual content, and conversational strategies to uncover emotion flip triggers, (3) cross-modal attention networks to fuse text and visual information, and (4) trajectory prediction to anticipate future emotional states. Evaluated on a large-scale MESC dataset of 23,126 therapeutic samples spanning 7 emotions, 41 flip types, 15 scenarios, and 10 intervention strategies, MultiFlipFormer achieves a final weighted F1-score of 0.828 across tasks. This framework provides a clear and reliable approach that can genuinely support therapy and help in planning better interventions.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

20203 - Telecommunications

Project

<a href="/en/project/VJ02010019" target="_blank" >VJ02010019: Tools for Handwriting fORensics</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

2025 17th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT)

ISBN

979-8-3315-7675-2

ISSN

—

e-ISSN

2157-023X

Number of pages

6

Pages from-to

187-192

Publisher name

IEEE

Place of publication

Florence, Italy

Event location

Florence, Italy

Event date

Nov 3, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—