Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F24%3A10492898" target="_blank" >RIV/00216208:11320/24:10492898 - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2024.alvr-1.9.pdf" target="_blank" >https://aclanthology.org/2024.alvr-1.9.pdf</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

Popis výsledku v původním jazyce

Current vision-language models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives replace terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding ac

Název v anglickém jazyce

Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

Popis výsledku anglicky

Current vision-language models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives replace terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding ac

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

The 3rd Workshop on Advances in Language and Vision Research: Proceedings of the Workshop

ISBN

979-8-89176-153-7

ISSN

—

e-ISSN

—

Počet stran výsledku

14

Strana od-do

102-115

Název nakladatele

Association for Computational Linguistics (ACL)

Místo vydání

Kerrville, TX, USA

Místo konání akce

Bangkok, Thailand

Datum konání akce

16. 8. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—