Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F24%3A00376304" target="_blank" >RIV/68407700:21730/24:00376304 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1109/IROS58592.2024.10802160" target="_blank" >https://doi.org/10.1109/IROS58592.2024.10802160</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/IROS58592.2024.10802160" target="_blank" >10.1109/IROS58592.2024.10802160</a>
Alternative languages
Result language
angličtina
Original language name
Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks
Original language description
In this work, we focus on unsupervised vision language-action mapping in the area of robotic manipulation. Recently, multiple approaches employing pre-trained large language and vision models have been proposed for this task. However, they are computationally demanding and require careful fine-tuning of the produced outputs. A more lightweight alternative would be the implementation of multimodal Vari ational Autoencoders (VAEs) which can extract the latent features of the data and integrate them into a joint repre sentation, as has been demonstrated mostly on image-image or image-text data for the state-of-the-art models. Here we explore whether and how can multimodal VAEs be employed in unsupervised robotic manipulation tasks in a simulated environment. Based on the obtained results, we propose a model-invariant training alternative that improves the models’ performance in a simulator by up to 55 %. Moreover, we systematically evaluate the challenges raised by the individual tasks such as object or robot position variability, number of distractors or the task length. Our work thus also sheds light on the potential benefits and limitations of using the current multimodal VAEs for unsupervised learning of robotic motion trajectories based on vision and language.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)
ISBN
979-8-3503-7770-5
ISSN
2153-0858
e-ISSN
2153-0866
Number of pages
7
Pages from-to
12522-12528
Publisher name
IEEE
Place of publication
Piscataway
Event location
Abu Dhabi
Event date
Oct 14, 2024
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
001433985300649