Learning to Manipulate Tools by Aligning Simulation to Video Demonstration
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21460%2F22%3A00354057" target="_blank" >RIV/68407700:21460/22:00354057 - isvavai.cz</a>
Alternative codes found
RIV/68407700:21730/22:00354057
Result on the web
<a href="https://doi.org/10.1109/LRA.2021.3127238" target="_blank" >https://doi.org/10.1109/LRA.2021.3127238</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/LRA.2021.3127238" target="_blank" >10.1109/LRA.2021.3127238</a>
Alternative languages
Result language
angličtina
Original language name
Learning to Manipulate Tools by Aligning Simulation to Video Demonstration
Original language description
A seamless integration of robots into human environments requires robots to learn how to use existing human tools. Current approaches for learning tool manipulation skills mostly rely on expert demonstrations provided in the target robot environment, for example, by manually guiding the robot manipulator or by teleoperation. In this work, we introduce an automated approach that replaces an expert demonstration with a Youtube video for learning a tool manipulation strategy. The main contributions are twofold. First, we design an alignment procedure that aligns the simulated environment with the real-world scene observed in the video. This is formulated as an optimization problem that finds a spatial alignment of the tool trajectory to maximize the sparse goal reward given by the environment. Second, we describe an imitation learning approach that focuses on the trajectory of the tool rather than the motion of the human. For this we combine reinforcement learning with an optimization procedure to find a control policy and the placement of the robot based on the tool motion in the aligned environment. We demonstrate the proposed approach on spade, scythe and hammer tools in simulation, and show the effectiveness of the trained policy for the spade on a real Franka Emika Panda robot demonstration.
Czech name
—
Czech description
—
Classification
Type
J<sub>imp</sub> - Article in a specialist periodical, which is included in the Web of Science database
CEP classification
—
OECD FORD branch
20204 - Robotics and automatic control
Result continuities
Project
<a href="/en/project/EF15_003%2F0000468" target="_blank" >EF15_003/0000468: Intelligent Machine Perception</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach
Others
Publication year
2022
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Name of the periodical
IEEE Robotics and Automation Letters
ISSN
2377-3766
e-ISSN
2377-3766
Volume of the periodical
7
Issue of the periodical within the volume
1
Country of publishing house
US - UNITED STATES
Number of pages
8
Pages from-to
438-445
UT code for WoS article
000724477700004
EID of the result in the Scopus database
2-s2.0-85119417079