Video–text retrieval via multi-modal masked transformer and adaptive attribute-aware graph convolutional network

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3ACCYPWNRV" target="_blank" >RIV/00216208:11320/25:CCYPWNRV - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85182811160&doi=10.1007%2fs00530-023-01205-8&partnerID=40&md5=f43954a8e2452cba63564609df02082a" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85182811160&doi=10.1007%2fs00530-023-01205-8&partnerID=40&md5=f43954a8e2452cba63564609df02082a</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s00530-023-01205-8" target="_blank" >10.1007/s00530-023-01205-8</a>

Result language

angličtina

Original language name

Video–text retrieval via multi-modal masked transformer and adaptive attribute-aware graph convolutional network

Original language description

Despite significant advancements in deep learning-based video–text retrieval methods, three challenges persist: the alignment of fine-grained semantic information from text and video, ensuring that the obtained textual and video feature representations capture primary semantic information while maintaining good discriminability, and measuring the semantic similarity between different instances. To tackle these issues, we introduce an end-to-end video–text retrieval framework which exploit Multi-Modal Masked Transformer and Adaptive Attribute-Aware Graph Convolutional Network (M 3 Trans-A 3 GCN). Specifically, the features extracted from videos and texts are fed into M 3 Trans to jointly integrate the multi-modal content and mask irrelevant multi-modal context. Subsequently, a novel GCN with an adaptive correlation matrix (i.e., A 3 GCN) is constructed to obtain discriminative video representation for video–text retrieval. To better measure the semantic similarity between video–text pairs during training, we propose a novel Text-semantic-guided Multi-Modal Cross-Entropy (TMCE) loss function. Here, the similarity between different video–text pairs within a batch is computed based on the features of the corresponding text rather than their instance labels. Comprehensive experimental results on three benchmark datasets, MSR-VTT, MSVD and LSMDC, demonstrate the superiority of M 3 Trans-A 3 GCN, compared with the state-of-the-art methods in video–text retrieval. © 2024, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.

Czech name

—

Czech description

—

Type

J_SC - Article in a specialist periodical, which is included in the SCOPUS database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Multimedia Systems

ISSN

0942-4962

e-ISSN

—

Volume of the periodical

30

Issue of the periodical within the volume

1

Country of publishing house

US - UNITED STATES

Number of pages

12

Pages from-to

1-12

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-85182811160