Joint-Dataset Learning and Cross-Consistent Regularization for Text-to-Motion Retrieval

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14330%2F25%3A00144684" target="_blank" >RIV/00216224:14330/25:00144684 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1145/3744565" target="_blank" >https://doi.org/10.1145/3744565</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3744565" target="_blank" >10.1145/3744565</a>

Result language

angličtina

Original language name

Joint-Dataset Learning and Cross-Consistent Regularization for Text-to-Motion Retrieval

Original language description

Pose-estimation methods enable extracting human motion from common videos in the structured form of 3D skeleton sequences. Despite great application opportunities, effective content-based access to such spatio-temporal motion data is a challenging problem. In this paper, we focus on the recently introduced text-motion retrieval tasks, which aim to search for database motions that are the most relevant to a specified natural-language textual description (text-to-motion) and vice-versa (motion-to-text). Despite recent efforts to explore these promising avenues, a primary challenge remains the insufficient data available to train robust text-motion models effectively. To address this issue, we propose to investigate joint-dataset learning - where we train on multiple text-motion datasets simultaneously - together with the introduction of a Cross-Consistent Contrastive Loss function (CCCL), which regularizes the learned text-motion common space by imposing uni-modal constraints that augment the representation ability of the trained network. To learn a proper motion representation, we also introduce a transformer-based motion encoder, called MoT++, which employs spatio-temporal attention to process skeleton data sequences. We demonstrate the benefits of the proposed approaches on the widely-used KIT Motion-Language and HumanML3D datasets, including also some results on the recent Motion-X dataset. We perform detailed experimentation on joint-dataset learning and cross-dataset scenarios, showing the effectiveness of each introduced module in a carefully conducted ablation study and, in turn, pointing out the limitations of state-of-the-art methods. The code for reproducing our results is available here: https://github.com/mesnico/MOTpp.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10200 - Computer and information sciences

Project

<a href="/en/project/VK01010147" target="_blank" >VK01010147: Automated digital data forensics lab for complex crime detection</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

ACM Transactions on Multimedia Computing, Communications, and Applications

ISSN

1551-6857

e-ISSN

—

Volume of the periodical

21

Issue of the periodical within the volume

10

Country of publishing house

US - UNITED STATES

Number of pages

24

Pages from-to

1-24

UT code for WoS article

001617271400001

EID of the result in the Scopus database

2-s2.0-105019646199