Infusing fine-grained visual knowledge to Vision-Language Models
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F25%3A00387296" target="_blank" >RIV/68407700:21230/25:00387296 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1109/ICCVW69036.2025.00445" target="_blank" >https://doi.org/10.1109/ICCVW69036.2025.00445</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/ICCVW69036.2025.00445" target="_blank" >10.1109/ICCVW69036.2025.00445</a>
Alternative languages
Result language
angličtina
Original language name
Infusing fine-grained visual knowledge to Vision-Language Models
Original language description
Large-scale contrastive pre-training produces powerful Vision-and-Language Models (VLMs) capable of generating representations (embeddings) effective for a wide variety of visual and multimodal tasks. However, these pretrained embeddings remain suboptimal for fine-grained open-set visual retrieval, where state-of-the-art results require fine-tuning the vision encoder using annotated domain-specific samples. Naively performing such fine-tuning typically leads to catastrophic forgetting, severely diminishing the model's general-purpose visual and cross-modal capabilities. In this work, we propose a fine-tuning method explicitly designed to achieve optimal balance between fine-grained domain adaptation and retention of the pretrained VLM's broad multimodal knowledge. Drawing inspiration from continual learning literature, we systematically analyze standard regularization techniques aimed at knowledge retention and propose an efficient and effective combination strategy. Additionally, we address the commonly overlooked yet critical aspects of validation set design and hyperparameter tuning to ensure reproducibility and robust generalization across datasets and pretrained models. We extensively evaluate our method on both fine-grained and coarse-grained image-image and image-text retrieval benchmarks. Our approach consistently achieves strong results, notably retaining the visual-text alignment without utilizing any text data or the original text encoder during fine-tuning. Code and model checkpoints: https://github.com/nikosips/infusing.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
S - Specificky vyzkum na vysokych skolach<br>I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2025
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
ICCVW2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
ISBN
979-8-3315-8989-9
ISSN
2473-9936
e-ISSN
2473-9944
Number of pages
10
Pages from-to
4285-4294
Publisher name
IEEE Communications Society
Place of publication
Anchorage
Event location
Honolulu
Event date
Oct 19, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—