All

What are you looking for?

All
Projects
Results
Organizations

Quick search

  • Projects supported by TA ČR
  • Excellent projects
  • Projects with the highest public support
  • Current projects

Smart search

  • That is how I find a specific +word
  • That is how I leave the -word out of the results
  • “That is how I can find the whole phrase”

Meta-Personalizing Vision-Language Models to Find Named Instances in Video

The result's identifiers

  • Result code in IS VaVaI

    <a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F23%3A00372042" target="_blank" >RIV/68407700:21730/23:00372042 - isvavai.cz</a>

  • Result on the web

    <a href="https://doi.org/10.1109/CVPR52729.2023.01833" target="_blank" >https://doi.org/10.1109/CVPR52729.2023.01833</a>

  • DOI - Digital Object Identifier

    <a href="http://dx.doi.org/10.1109/CVPR52729.2023.01833" target="_blank" >10.1109/CVPR52729.2023.01833</a>

Alternative languages

  • Result language

    angličtina

  • Original language name

    Meta-Personalizing Vision-Language Models to Find Named Instances in Video

  • Original language description

    Large-scale vision-language models (VLM) have shown impressive results for language-guided search applications. While these models allow category-level queries, they currently struggle with personalized searches for moments in a video where a specific object instance such as “My dog Biscuit” appears. We present the following three contributions to address this problem. First, we describe a method to meta-personalize a pre-trained VLM, i.e., learning how to learn to personalize a VLM at test time to search in video. Our method extends the VLM's token vocabulary by learning novel word embeddings specific to each instance. To capture only instance-specific features, we represent each instance embedding as a combination of shared and learned global category features. Second, we propose to learn such personalization without explicit human supervision. Our approach automatically identifies moments of named visual instances in video using transcripts and vision-language similarity in the VLM's embedding space. Finally, we introduce This-Is-My, a personal video instance retrieval benchmark. We evaluate our approach on This-Is-My and Deep-Fashion2 and show that we obtain a 15% relative improvement over the state of the art on the latter dataset.

  • Czech name

  • Czech description

Classification

  • Type

    D - Article in proceedings

  • CEP classification

  • OECD FORD branch

    10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Result continuities

  • Project

  • Continuities

    I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Others

  • Publication year

    2023

  • Confidentiality

    S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Data specific for result type

  • Article name in the collection

    Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  • ISBN

    979-8-3503-0130-4

  • ISSN

    1063-6919

  • e-ISSN

    2575-7075

  • Number of pages

    10

  • Pages from-to

    19123-19132

  • Publisher name

    IEEE Computer Society

  • Place of publication

    USA

  • Event location

    Vancouver

  • Event date

    Jun 18, 2023

  • Type of event by nationality

    WRD - Celosvětová akce

  • UT code for WoS article

    001062531303042