Multimodal Emotion Recognition Through a Hybrid CNN-GCN Model Integrating Neuroimaging and Physiological Data

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21460%2F25%3A00384262" target="_blank" >RIV/68407700:21460/25:00384262 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1007/978-3-031-86323-3_47" target="_blank" >https://doi.org/10.1007/978-3-031-86323-3_47</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-031-86323-3_47" target="_blank" >10.1007/978-3-031-86323-3_47</a>

Result language

angličtina

Original language name

Multimodal Emotion Recognition Through a Hybrid CNN-GCN Model Integrating Neuroimaging and Physiological Data

Original language description

Emotion recognition plays an important role in understanding human affective states and holds significant potential in diagnostic and therapeutic applications. This study presents a novel hybrid model that integrates convolutional neural networks and graph convolutional networks to improve multimodal emotion recognition by leveraging neuroimaging and physiological data. Specifically, functional magnetic resonance imaging, photoplethysmography, and respiratory signals were utilized to predict emotional valence. The proposed model processes fMRI data to capture spatial and temporal brain activity, while physiological signals are transformed into Gramian Angular Fields. to extract temporal features. The model’s performance was evaluated using a Leave-One-Subject-Out cross-validation strategy on the dataset provided by the ICBHI 2024 Scientific Challenge, achieving a competition score of 0.3314. The experimental results demonstrated an average accuracy of 70.42% ± 0.06% in the valence class prediction task and an average of the mean absolute error of 1.74 ± 0.46 in the valence rating prediction. Our hybrid model performance showed an accuracy of 83.33% in the classification task.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of ICBHI 2024

ISBN

978-3-031-86322-6

ISSN

1680-0737

e-ISSN

1433-9277

Number of pages

7

Pages from-to

399-405

Publisher name

Springer Nature

Place of publication

—

Event location

Tainan

Event date

Oct 30, 2024

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

001480887600047