Multimodal Emotion Recognition Through a Hybrid CNN-GCN Model Integrating Neuroimaging and Physiological Data

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21460%2F25%3A00384262" target="_blank" >RIV/68407700:21460/25:00384262 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1007/978-3-031-86323-3_47" target="_blank" >https://doi.org/10.1007/978-3-031-86323-3_47</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-031-86323-3_47" target="_blank" >10.1007/978-3-031-86323-3_47</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Multimodal Emotion Recognition Through a Hybrid CNN-GCN Model Integrating Neuroimaging and Physiological Data

Popis výsledku v původním jazyce

Emotion recognition plays an important role in understanding human affective states and holds significant potential in diagnostic and therapeutic applications. This study presents a novel hybrid model that integrates convolutional neural networks and graph convolutional networks to improve multimodal emotion recognition by leveraging neuroimaging and physiological data. Specifically, functional magnetic resonance imaging, photoplethysmography, and respiratory signals were utilized to predict emotional valence. The proposed model processes fMRI data to capture spatial and temporal brain activity, while physiological signals are transformed into Gramian Angular Fields. to extract temporal features. The model’s performance was evaluated using a Leave-One-Subject-Out cross-validation strategy on the dataset provided by the ICBHI 2024 Scientific Challenge, achieving a competition score of 0.3314. The experimental results demonstrated an average accuracy of 70.42% ± 0.06% in the valence class prediction task and an average of the mean absolute error of 1.74 ± 0.46 in the valence rating prediction. Our hybrid model performance showed an accuracy of 83.33% in the classification task.

Název v anglickém jazyce

Multimodal Emotion Recognition Through a Hybrid CNN-GCN Model Integrating Neuroimaging and Physiological Data

Popis výsledku anglicky

Emotion recognition plays an important role in understanding human affective states and holds significant potential in diagnostic and therapeutic applications. This study presents a novel hybrid model that integrates convolutional neural networks and graph convolutional networks to improve multimodal emotion recognition by leveraging neuroimaging and physiological data. Specifically, functional magnetic resonance imaging, photoplethysmography, and respiratory signals were utilized to predict emotional valence. The proposed model processes fMRI data to capture spatial and temporal brain activity, while physiological signals are transformed into Gramian Angular Fields. to extract temporal features. The model’s performance was evaluated using a Leave-One-Subject-Out cross-validation strategy on the dataset provided by the ICBHI 2024 Scientific Challenge, achieving a competition score of 0.3314. The experimental results demonstrated an average accuracy of 70.42% ± 0.06% in the valence class prediction task and an average of the mean absolute error of 1.74 ± 0.46 in the valence rating prediction. Our hybrid model performance showed an accuracy of 83.33% in the classification task.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of ICBHI 2024

ISBN

978-3-031-86322-6

ISSN

1680-0737

e-ISSN

1433-9277

Počet stran výsledku

7

Strana od-do

399-405

Název nakladatele

Springer Nature

Místo vydání

—

Místo konání akce

Tainan

Datum konání akce

30. 10. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

001480887600047