Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F24%3A00376304" target="_blank" >RIV/68407700:21730/24:00376304 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1109/IROS58592.2024.10802160" target="_blank" >https://doi.org/10.1109/IROS58592.2024.10802160</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/IROS58592.2024.10802160" target="_blank" >10.1109/IROS58592.2024.10802160</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks

Popis výsledku v původním jazyce

In this work, we focus on unsupervised vision language-action mapping in the area of robotic manipulation. Recently, multiple approaches employing pre-trained large language and vision models have been proposed for this task. However, they are computationally demanding and require careful fine-tuning of the produced outputs. A more lightweight alternative would be the implementation of multimodal Vari ational Autoencoders (VAEs) which can extract the latent features of the data and integrate them into a joint repre sentation, as has been demonstrated mostly on image-image or image-text data for the state-of-the-art models. Here we explore whether and how can multimodal VAEs be employed in unsupervised robotic manipulation tasks in a simulated environment. Based on the obtained results, we propose a model-invariant training alternative that improves the models’ performance in a simulator by up to 55 %. Moreover, we systematically evaluate the challenges raised by the individual tasks such as object or robot position variability, number of distractors or the task length. Our work thus also sheds light on the potential benefits and limitations of using the current multimodal VAEs for unsupervised learning of robotic motion trajectories based on vision and language.

Název v anglickém jazyce

Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks

Popis výsledku anglicky

In this work, we focus on unsupervised vision language-action mapping in the area of robotic manipulation. Recently, multiple approaches employing pre-trained large language and vision models have been proposed for this task. However, they are computationally demanding and require careful fine-tuning of the produced outputs. A more lightweight alternative would be the implementation of multimodal Vari ational Autoencoders (VAEs) which can extract the latent features of the data and integrate them into a joint repre sentation, as has been demonstrated mostly on image-image or image-text data for the state-of-the-art models. Here we explore whether and how can multimodal VAEs be employed in unsupervised robotic manipulation tasks in a simulated environment. Based on the obtained results, we propose a model-invariant training alternative that improves the models’ performance in a simulator by up to 55 %. Moreover, we systematically evaluate the challenges raised by the individual tasks such as object or robot position variability, number of distractors or the task length. Our work thus also sheds light on the potential benefits and limitations of using the current multimodal VAEs for unsupervised learning of robotic motion trajectories based on vision and language.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)

ISBN

979-8-3503-7770-5

ISSN

2153-0858

e-ISSN

2153-0866

Počet stran výsledku

7

Strana od-do

12522-12528

Název nakladatele

IEEE

Místo vydání

Piscataway

Místo konání akce

Abu Dhabi

Datum konání akce

14. 10. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

001433985300649