Learning Object Manipulation Skills from Video via Approximate Differentiable Physics

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F22%3A00364136" target="_blank" >RIV/68407700:21730/22:00364136 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1109/IROS47612.2022.9982084" target="_blank" >https://doi.org/10.1109/IROS47612.2022.9982084</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/IROS47612.2022.9982084" target="_blank" >10.1109/IROS47612.2022.9982084</a>

Result language

angličtina

Original language name

Learning Object Manipulation Skills from Video via Approximate Differentiable Physics

Original language description

We aim to teach robots to perform simple object manipulation tasks by watching a single video demonstration. Towards this goal, we propose an optimization approach that outputs a coarse and temporally evolving 3D scene to mimic the action demonstrated in the input video. Similar to previous work, a differentiable renderer ensures perceptual fidelity between the 3D scene and the 2D video. Our key novelty lies in the inclusion of a differentiable approach to solve a set of Ordinary Differential Equations (ODEs) that allows us to approximately model laws of physics such as gravity, friction, and hand-object or object-object interactions. This not only enables us to dramatically improve the quality of estimated hand and object states, but also produces physically admissible trajectories that can be directly translated to a robot without the need for costly reinforcement learning. We evaluate our approach on a 3D reconstruction task that consists of 54 video demonstrations sourced from 9 actions such as pull something from right to left or put something in front of something. Our approach improves over previous state-of-the-art by almost 30%, demonstrating superior quality on especially challenging actions involving physical interactions of two objects such as put something onto something. Finally, we showcase the learned skills on a Franka Emika Panda robot.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/EF15_003%2F0000468" target="_blank" >EF15_003/0000468: Intelligent Machine Perception</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2022

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Intelligent Robots and Systems (IROS), 2022 IEEE/RSJ International Conference on

ISBN

978-1-6654-7927-1

ISSN

2153-0858

e-ISSN

2153-0866

Number of pages

8

Pages from-to

7375-7382

Publisher name

IEEE

Place of publication

Piscataway

Event location

Kyoto

Event date

Oct 23, 2022

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

000909405300038