Multi-Task Learning of Object States and State-Modifying Actions From Web Videos

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F24%3A00380611" target="_blank" >RIV/68407700:21730/24:00380611 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1109/TPAMI.2024.3362288" target="_blank" >https://doi.org/10.1109/TPAMI.2024.3362288</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/TPAMI.2024.3362288" target="_blank" >10.1109/TPAMI.2024.3362288</a>

Result language

angličtina

Original language name

Multi-Task Learning of Object States and State-Modifying Actions From Web Videos

Original language description

We aim to learn to temporally localize object state changes and the corresponding state-modifying actions by observing people interacting with objects in long uncurated web videos. We introduce three principal contributions. First, we develop a self-supervised model for jointly learning state-modifying actions together with the corresponding object states from an uncurated set of videos from the Internet. The model is self-supervised by the causal ordering signal, i.e., initial object state -> manipulating action -> end state. Second, we explore alternative multi-task network architectures and identify a model that enables efficient joint learning of multiple object states and actions, such as pouring water and pouring coffee, together. Third, we collect a new dataset, named ChangeIt, with more than 2600 hours of video and 34 thousand changes of object states. We report results on an existing instructional video dataset COIN as well as our new large-scale ChangeIt dataset containing tens of thousands of long uncurated web videos depicting various interactions such as hole drilling, cream whisking, or paper plane folding. We show that our multi-task model achieves a relative improvement of 40% over the prior methods and significantly outperforms both image-based and video-based zero-shot models for this problem.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/EF15_003%2F0000468" target="_blank" >EF15_003/0000468: Intelligent Machine Perception</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

IEEE Transactions on Pattern Analysis and Machine Intelligence

ISSN

0162-8828

e-ISSN

1939-3539

Volume of the periodical

46

Issue of the periodical within the volume

7

Country of publishing house

US - UNITED STATES

Number of pages

17

Pages from-to

5114-5130

UT code for WoS article

001240147800027

EID of the result in the Scopus database

2-s2.0-85184804699