GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F24%3A00376353" target="_blank" >RIV/68407700:21730/24:00376353 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1109/CVPR52733.2024.00627" target="_blank" >https://doi.org/10.1109/CVPR52733.2024.00627</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/CVPR52733.2024.00627" target="_blank" >10.1109/CVPR52733.2024.00627</a>
Alternative languages
Result language
angličtina
Original language name
GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
Original language description
We address the task of generating temporally consistent and physically plausible images of actions and object state transformations. Given an input image and a text prompt describing the targeted transformation, our generated images preserve the environment and transform objects in the initial image. Our contributions are threefold. First, we leverage a large body of instructional videos and automatically mine a dataset of triplets of consecutive frames corresponding to initial object states, actions, and resulting object transformations. Second, equipped with this data, we develop and train a conditioned diffusion model dubbed GenHowTo. Third, we evaluate GenHowTo on a variety of objects and actions and show superior performance compared to existing methods. In particular, we introduce a quantitative evaluation where GenHowTo achieves 88% and 74% on seen and unseen interaction categories, respectively, outperforming prior work by a large margin.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
The Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024
ISBN
979-8-3503-5300-6
ISSN
1063-6919
e-ISSN
2575-7075
Number of pages
11
Pages from-to
6561-6571
Publisher name
IEEE Xplore
Place of publication
—
Event location
Seattle , WA
Event date
Jun 17, 2024
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
001322555906092