Embodied Reasoning for Discovering Object Properties via Manipulation

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F21%3A00352665" target="_blank" >RIV/68407700:21230/21:00352665 - isvavai.cz</a>

Nalezeny alternativní kódy

RIV/68407700:21730/21:00352665

Výsledek na webu

<a href="https://doi.org/10.1109/ICRA48506.2021.9561212" target="_blank" >https://doi.org/10.1109/ICRA48506.2021.9561212</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/ICRA48506.2021.9561212" target="_blank" >10.1109/ICRA48506.2021.9561212</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Embodied Reasoning for Discovering Object Properties via Manipulation

Popis výsledku v původním jazyce

In this paper, we present an integrated system that includes reasoning from visual and natural language inputs, action and motion planning, executing tasks by a robotic arm, manipulating objects, and discovering their properties. A vision to action module recognises the scene with objects and their attributes and analyses enquiries formulated in natural language. It performs multi-modal reasoning and generates a sequence of simple actions that can be executed by a robot. The scene model and action sequence are sent to a planning and execution module that generates a motion plan with collision avoidance, simulates the actions, and executes them. We use synthetic data to train various components of the system and test on a real robot to show the generalization capabilities. We focus on a tabletop scenario with objects that can be grasped by our embodied agent i.e. a 7DoF manipulator with a two-finger gripper. We evaluate the agent on 60 representative queries repeated 3 times (e.g., ’Check what is on the other side of the soda can’) concerning different objects and tasks in the scene. We perform experiments in a simulated and real environment and report the success rate for various components of the system. Our system achieves up to 80.6% success rate on challenging scenes and queries. We also analyse and discuss the challenges that such an intelligent embodied system faces.

Název v anglickém jazyce

Embodied Reasoning for Discovering Object Properties via Manipulation

Popis výsledku anglicky

In this paper, we present an integrated system that includes reasoning from visual and natural language inputs, action and motion planning, executing tasks by a robotic arm, manipulating objects, and discovering their properties. A vision to action module recognises the scene with objects and their attributes and analyses enquiries formulated in natural language. It performs multi-modal reasoning and generates a sequence of simple actions that can be executed by a robot. The scene model and action sequence are sent to a planning and execution module that generates a motion plan with collision avoidance, simulates the actions, and executes them. We use synthetic data to train various components of the system and test on a real robot to show the generalization capabilities. We focus on a tabletop scenario with objects that can be grasped by our embodied agent i.e. a 7DoF manipulator with a two-finger gripper. We evaluate the agent on 60 representative queries repeated 3 times (e.g., ’Check what is on the other side of the soda can’) concerning different objects and tasks in the scene. We perform experiments in a simulated and real environment and report the success rate for various components of the system. Our system achieves up to 80.6% success rate on challenging scenes and queries. We also analyse and discuss the challenges that such an intelligent embodied system faces.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

20204 - Robotics and automatic control

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

IEEE International Conference on Robotics and Automation (ICRA)

ISBN

978-1-7281-9077-8

ISSN

1050-4729

e-ISSN

2577-087X

Počet stran výsledku

7

Strana od-do

10139-10145

Název nakladatele

IEEE Xplore

Místo vydání

—

Místo konání akce

Xi’an

Datum konání akce

30. 5. 2021

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

000771405403013