ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F25%3A00388523" target="_blank" >RIV/68407700:21730/25:00388523 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1109/CVPR52734.2025.02555" target="_blank" >https://doi.org/10.1109/CVPR52734.2025.02555</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/CVPR52734.2025.02555" target="_blank" >10.1109/CVPR52734.2025.02555</a>

Jazyk výsledku

angličtina

Název v původním jazyce

ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions

Popis výsledku v původním jazyce

The goal of this work is to generate step-by-step visual in structions in the form of a sequence of images, given an in put image that provides the scene context and the sequence of textual instructions. This is a challenging problem as it requires generating multi-step image sequences to achieve a complex goal while being grounded in a specific environment. Part of the challenge stems from the lack of large scale training data for this problem. The contribution of this work is thus three-fold. First, we introduce an automatic approach for collecting large step-by-step visual in struction training data from instructional videos. We apply this approach to one million videos and create a large-scale, high-quality dataset of 0.6M sequences of image-text pairs. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image. Third, we evaluate the generated image sequences across three dimensions of accuracy (step, scene, and task) and show our model achieves state-of-the-art results on all of them. Our code, dataset, and trained models are publicly available.

Název v anglickém jazyce

ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions

Popis výsledku anglicky

The goal of this work is to generate step-by-step visual in structions in the form of a sequence of images, given an in put image that provides the scene context and the sequence of textual instructions. This is a challenging problem as it requires generating multi-step image sequences to achieve a complex goal while being grounded in a specific environment. Part of the challenge stems from the lack of large scale training data for this problem. The contribution of this work is thus three-fold. First, we introduce an automatic approach for collecting large step-by-step visual in struction training data from instructional videos. We apply this approach to one million videos and create a large-scale, high-quality dataset of 0.6M sequences of image-text pairs. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image. Third, we evaluate the generated image sequences across three dimensions of accuracy (step, scene, and task) and show our model achieves state-of-the-art results on all of them. Our code, dataset, and trained models are publicly available.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

R - Projekt Ramcoveho programu EK

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

ISBN

979-8-3315-4364-8

ISSN

1063-6919

e-ISSN

2575-7075

Počet stran výsledku

11

Strana od-do

27435-27445

Název nakladatele

IEEE Computer Society

Místo vydání

Los Alamitos

Místo konání akce

Nashville

Datum konání akce

11. 6. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

001601181100317