Test-time Training for Matching-based Video Object Segmentation

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F23%3A00370562" target="_blank" >RIV/68407700:21230/23:00370562 - isvavai.cz</a>

Result on the web

<a href="https://openreview.net/pdf?id=9QsdPQlWiE" target="_blank" >https://openreview.net/pdf?id=9QsdPQlWiE</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Test-time Training for Matching-based Video Object Segmentation

Original language description

The video object segmentation (VOS) task involves the segmentation of an object over time based on a single initial mask. Current state-of-the-art approaches use a memory of previously processed frames and rely on matching to estimate segmentation masks of subsequent frames. Lacking any adaptation mechanism, such methods are prone to test-time distribution shifts. This work focuses on matching-based VOS under distribution shifts such as video corruptions, stylization, and sim-to-real transfer. We explore test-time training strategies that are agnostic to the specific task as well as strategies that are designed specifically for VOS. This includes a variant based on mask cycle consistency tailored to matching-based VOS methods. The experimental results on common benchmarks demonstrate that the proposed test-time training yields significant improvements in performance. In particular for the sim-to-real scenario and despite using only a single test video, our approach manages to recover a substantial portion of the performance gain achieved through training on real videos. Additionally, we introduce DAVIS-C, an augmented version of the popular DAVIS test set, featuring extreme distribution shifts like image-/video-level corruptions and stylizations. Our results illustrate that test-time training enhances performance even in these challenging cases.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/GM21-28830M" target="_blank" >GM21-28830M: Learning Universal Visual Representation with Limited Supervision</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Advances in Neural Information Processing Systems 36 (NeurIPS 2023)

ISBN

—

ISSN

1049-5258

e-ISSN

—

Number of pages

24

Pages from-to

—

Publisher name

Neural Information Processing Society

Place of publication

Montreal

Event location

New Orleans

Event date

Dec 10, 2023

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—