Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F24%3A10492898" target="_blank" >RIV/00216208:11320/24:10492898 - isvavai.cz</a>

Result on the web

<a href="https://aclanthology.org/2024.alvr-1.9.pdf" target="_blank" >https://aclanthology.org/2024.alvr-1.9.pdf</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

Original language description

Current vision-language models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives replace terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding ac

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

The 3rd Workshop on Advances in Language and Vision Research: Proceedings of the Workshop

ISBN

979-8-89176-153-7

ISSN

—

e-ISSN

—

Number of pages

14

Pages from-to

102-115

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

Kerrville, TX, USA

Event location

Bangkok, Thailand

Event date

Aug 16, 2024

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—