ResidualViT for Efficient Temporally Dense Video Encoding

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F25%3A00388532" target="_blank" >RIV/68407700:21730/25:00388532 - isvavai.cz</a>

Výsledek na webu

<a href="https://openaccess.thecvf.com/content/ICCV2025/papers/Soldan_ResidualViT_for_Efficient_Temporally_Dense_Video_Encoding_ICCV_2025_paper.pdf" target="_blank" >https://openaccess.thecvf.com/content/ICCV2025/papers/Soldan_ResidualViT_for_Efficient_Temporally_Dense_Video_Encoding_ICCV_2025_paper.pdf</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

ResidualViT for Efficient Temporally Dense Video Encoding

Popis výsledku v původním jazyce

Several video understanding tasks, such as natural language temporal video grounding, temporal activity localization, and audio description generation, require "temporally dense" reasoning over frames sampled at high temporal resolution. However, computing frame-level features for these tasks is computationally expensive given the temporal resolution requirements. In this paper, we make three contributions to reduce the cost of computing features for temporally dense tasks. First, we introduce a vision transformer (ViT) architecture, dubbed ResidualViT, that leverages the large temporal redundancy in videos to efficiently compute temporally dense frame-level features. Our architecture incorporates (i) learnable residual connections that ensure temporal consistency across consecutive frames and (ii) a token reduction module that enhances processing speed by selectively discarding temporally redundant information while reusing weights of a pretrained foundation model. Second, we propose a lightweight distillation strategy to approximate the frame-level features of the original foundation model. Finally, we evaluate our approach across four tasks and five datasets, in both zero-shot and fully supervised settings, demonstrating significant reductions in computational cost (up to ) and improvements in inference speed (up to faster), all while closely approximating the accuracy of the original foundation model.

Název v anglickém jazyce

ResidualViT for Efficient Temporally Dense Video Encoding

Popis výsledku anglicky

Several video understanding tasks, such as natural language temporal video grounding, temporal activity localization, and audio description generation, require "temporally dense" reasoning over frames sampled at high temporal resolution. However, computing frame-level features for these tasks is computationally expensive given the temporal resolution requirements. In this paper, we make three contributions to reduce the cost of computing features for temporally dense tasks. First, we introduce a vision transformer (ViT) architecture, dubbed ResidualViT, that leverages the large temporal redundancy in videos to efficiently compute temporally dense frame-level features. Our architecture incorporates (i) learnable residual connections that ensure temporal consistency across consecutive frames and (ii) a token reduction module that enhances processing speed by selectively discarding temporally redundant information while reusing weights of a pretrained foundation model. Second, we propose a lightweight distillation strategy to approximate the frame-level features of the original foundation model. Finally, we evaluate our approach across four tasks and five datasets, in both zero-shot and fully supervised settings, demonstrating significant reductions in computational cost (up to ) and improvements in inference speed (up to faster), all while closely approximating the accuracy of the original foundation model.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

N - Vyzkumna aktivita podporovana z neverejnych zdroju

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

ICCV2025: Proceedings of the International Conference on Computer Vision

ISBN

—

ISSN

1550-5499

e-ISSN

—

Počet stran výsledku

11

Strana od-do

22305-22315

Název nakladatele

IEEE Communications Society

Místo vydání

Anchorage

Místo konání akce

Honolulu

Datum konání akce

19. 10. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—