GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F24%3A00376353" target="_blank" >RIV/68407700:21730/24:00376353 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1109/CVPR52733.2024.00627" target="_blank" >https://doi.org/10.1109/CVPR52733.2024.00627</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/CVPR52733.2024.00627" target="_blank" >10.1109/CVPR52733.2024.00627</a>

Jazyk výsledku

angličtina

Název v původním jazyce

GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos

Popis výsledku v původním jazyce

We address the task of generating temporally consistent and physically plausible images of actions and object state transformations. Given an input image and a text prompt describing the targeted transformation, our generated images preserve the environment and transform objects in the initial image. Our contributions are threefold. First, we leverage a large body of instructional videos and automatically mine a dataset of triplets of consecutive frames corresponding to initial object states, actions, and resulting object transformations. Second, equipped with this data, we develop and train a conditioned diffusion model dubbed GenHowTo. Third, we evaluate GenHowTo on a variety of objects and actions and show superior performance compared to existing methods. In particular, we introduce a quantitative evaluation where GenHowTo achieves 88% and 74% on seen and unseen interaction categories, respectively, outperforming prior work by a large margin.

Název v anglickém jazyce

GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos

Popis výsledku anglicky

We address the task of generating temporally consistent and physically plausible images of actions and object state transformations. Given an input image and a text prompt describing the targeted transformation, our generated images preserve the environment and transform objects in the initial image. Our contributions are threefold. First, we leverage a large body of instructional videos and automatically mine a dataset of triplets of consecutive frames corresponding to initial object states, actions, and resulting object transformations. Second, equipped with this data, we develop and train a conditioned diffusion model dubbed GenHowTo. Third, we evaluate GenHowTo on a variety of objects and actions and show superior performance compared to existing methods. In particular, we introduce a quantitative evaluation where GenHowTo achieves 88% and 74% on seen and unseen interaction categories, respectively, outperforming prior work by a large margin.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

The Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024

ISBN

979-8-3503-5300-6

ISSN

1063-6919

e-ISSN

2575-7075

Počet stran výsledku

11

Strana od-do

6561-6571

Název nakladatele

IEEE Xplore

Místo vydání

—

Místo konání akce

Seattle , WA

Datum konání akce

17. 6. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

001322555906092