Meta-Personalizing Vision-Language Models to Find Named Instances in Video
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F23%3A00372042" target="_blank" >RIV/68407700:21730/23:00372042 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1109/CVPR52729.2023.01833" target="_blank" >https://doi.org/10.1109/CVPR52729.2023.01833</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/CVPR52729.2023.01833" target="_blank" >10.1109/CVPR52729.2023.01833</a>
Alternative languages
Result language
angličtina
Original language name
Meta-Personalizing Vision-Language Models to Find Named Instances in Video
Original language description
Large-scale vision-language models (VLM) have shown impressive results for language-guided search applications. While these models allow category-level queries, they currently struggle with personalized searches for moments in a video where a specific object instance such as “My dog Biscuit” appears. We present the following three contributions to address this problem. First, we describe a method to meta-personalize a pre-trained VLM, i.e., learning how to learn to personalize a VLM at test time to search in video. Our method extends the VLM's token vocabulary by learning novel word embeddings specific to each instance. To capture only instance-specific features, we represent each instance embedding as a combination of shared and learned global category features. Second, we propose to learn such personalization without explicit human supervision. Our approach automatically identifies moments of named visual instances in video using transcripts and vision-language similarity in the VLM's embedding space. Finally, we introduce This-Is-My, a personal video instance retrieval benchmark. We evaluate our approach on This-Is-My and Deep-Fashion2 and show that we obtain a 15% relative improvement over the state of the art on the latter dataset.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2023
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
ISBN
979-8-3503-0130-4
ISSN
1063-6919
e-ISSN
2575-7075
Number of pages
10
Pages from-to
19123-19132
Publisher name
IEEE Computer Society
Place of publication
USA
Event location
Vancouver
Event date
Jun 18, 2023
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
001062531303042