Embodied Reasoning for Discovering Object Properties via Manipulation

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F21%3A00352665" target="_blank" >RIV/68407700:21230/21:00352665 - isvavai.cz</a>

Alternative codes found

RIV/68407700:21730/21:00352665

Result on the web

<a href="https://doi.org/10.1109/ICRA48506.2021.9561212" target="_blank" >https://doi.org/10.1109/ICRA48506.2021.9561212</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/ICRA48506.2021.9561212" target="_blank" >10.1109/ICRA48506.2021.9561212</a>

Result language

angličtina

Original language name

Embodied Reasoning for Discovering Object Properties via Manipulation

Original language description

In this paper, we present an integrated system that includes reasoning from visual and natural language inputs, action and motion planning, executing tasks by a robotic arm, manipulating objects, and discovering their properties. A vision to action module recognises the scene with objects and their attributes and analyses enquiries formulated in natural language. It performs multi-modal reasoning and generates a sequence of simple actions that can be executed by a robot. The scene model and action sequence are sent to a planning and execution module that generates a motion plan with collision avoidance, simulates the actions, and executes them. We use synthetic data to train various components of the system and test on a real robot to show the generalization capabilities. We focus on a tabletop scenario with objects that can be grasped by our embodied agent i.e. a 7DoF manipulator with a two-finger gripper. We evaluate the agent on 60 representative queries repeated 3 times (e.g., ’Check what is on the other side of the soda can’) concerning different objects and tasks in the scene. We perform experiments in a simulated and real environment and report the success rate for various components of the system. Our system achieves up to 80.6% success rate on challenging scenes and queries. We also analyse and discuss the challenges that such an intelligent embodied system faces.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

20204 - Robotics and automatic control

Project

Result was created during the realization of more than one project. More information in the Projects tab.

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2021

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

IEEE International Conference on Robotics and Automation (ICRA)

ISBN

978-1-7281-9077-8

ISSN

1050-4729

e-ISSN

2577-087X

Number of pages

7

Pages from-to

10139-10145

Publisher name

IEEE Xplore

Place of publication

—

Event location

Xi’an

Event date

May 30, 2021

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

000771405403013