6D Object Pose Tracking in Internet Videos for Robotic Manipulation

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F25%3A00383843" target="_blank" >RIV/68407700:21230/25:00383843 - isvavai.cz</a>

Alternative codes found

RIV/68407700:21730/25:00383843

Result on the web

<a href="https://openreview.net/pdf?id=1CIUkpoata" target="_blank" >https://openreview.net/pdf?id=1CIUkpoata</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

6D Object Pose Tracking in Internet Videos for Robotic Manipulation

Original language description

Weseektoextract a temporally consistent 6D pose trajectory of a manipulated object from an Internet instructional video. This is a challenging set-up for current 6D pose estimation methods due to uncontrolled capturing conditions, subtle but dynamic object motions, and the fact that the exact mesh of the manipulated object is not known. To address these challenges, we present the following contributions. First, we develop a new method that estimates the 6D pose of any object in the input image without prior knowledge of the object itself. The method proceeds by (i) retrieving a CAD model similar to the depicted object from a large-scale model database, (ii) 6D aligning the retrieved CAD model with the input image, and (iii) grounding the absolute scale of the object with respect to the scene. Second, we extract smooth 6D object trajectories from Internet videos by carefully tracking the detected objects across video frames. The extracted object trajectories are then retargeted via trajectory optimization into the configuration space of a robotic manipulator. Third, we thoroughly evaluate and ablate our 6D pose estimation method on YCB-V and HOPE-Video datasets as well as a new dataset of instructional videos manually annotated with approximate 6D object trajectories. We demonstrate significant improvements over existing state-of-the-art RGB 6D pose estimation methods. Finally, we show that the 6D object motion estimated from Internet videos can be transferred to a 7-axis robotic manipulator both in a virtual simulator as well as in a real world set-up. We also successfully apply our method to egocentric videos taken from the EPIC-KITCHENS dataset, demonstrating potential for Embodied AI applications.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

LEARNING REPRESENTATIONS. INTERNATIONAL CONFERENCE. 13TH 2025. (ICLR 2025)

ISBN

9798331320850

ISSN

—

e-ISSN

—

Number of pages

28

Pages from-to

1634-1661

Publisher name

International Conference on Learning Representations

Place of publication

—

Event location

Singapore EXPO

Event date

Apr 24, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—