POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F23%3A00371012" target="_blank" >RIV/68407700:21230/23:00371012 - isvavai.cz</a>

Alternative codes found

RIV/68407700:21730/23:00371012

Result on the web

<a href="https://proceedings.neurips.cc/paper_files/paper/2023/file/9e30acdeff572463c1db9b7de59de64c-Paper-Conference.pdf" target="_blank" >https://proceedings.neurips.cc/paper_files/paper/2023/file/9e30acdeff572463c1db9b7de59de64c-Paper-Conference.pdf</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.48550/arXiv.2401.09413" target="_blank" >10.48550/arXiv.2401.09413</a>

Result language

angličtina

Original language name

POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images

Original language description

We describe an approach to predict open-vocabulary 3D semantic voxel occupancy map from input 2D images with the objective of enabling 3D grounding, segmentation and retrieval of free-form language queries. This is a challenging problem because of the 2D-3D ambiguity and the open-vocabulary nature of the target tasks, where obtaining annotated training data in 3D is difficult. The con- tributions of this work are three-fold. First, we design a new model architecture for open-vocabulary 3D semantic occupancy prediction. The architecture consists of a 2D-3D encoder together with occupancy prediction and 3D-language heads. The output is a dense voxel map of 3D grounded language embeddings enabling a range of open-vocabulary tasks. Second, we develop a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds, and enables training the proposed architecture using a strong pre-trained vision-language model without the need for any 3D manual language annotations. Finally, we demonstrate quantitatively the strengths of the proposed model on several open-vocabulary tasks: Zero-shot 3D semantic segmentation using existing datasets; 3D grounding and retrieval of free-form language queries, using a small dataset that we propose as an extension of nuScenes.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/EF15_003%2F0000468" target="_blank" >EF15_003/0000468: Intelligent Machine Perception</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Advances in Neural Information Processing Systems 36 (NeurIPS 2023)

ISBN

—

ISSN

1049-5258

e-ISSN

—

Number of pages

13

Pages from-to

50545-50557

Publisher name

Neural Information Processing Society

Place of publication

Montreal

Event location

New Orleans

Event date

Dec 10, 2023

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

001227224008029