Probing the Role of Positional Information in Vision-Language Models

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F22%3A10457088" target="_blank" >RIV/00216208:11320/22:10457088 - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2022.findings-naacl.77" target="_blank" >https://aclanthology.org/2022.findings-naacl.77</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2022.findings-naacl.77" target="_blank" >10.18653/v1/2022.findings-naacl.77</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Probing the Role of Positional Information in Vision-Language Models

Popis výsledku v původním jazyce

In most Vision-Language models (VL), the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we probe the use of the PI in the representation and study its effect on Visual Question Answering. We show that the model is not capable of leveraging the PI for the image-text matching task on a challenge set where only position differs. Yet, our experiments with probing confirm that the PI is indeed present in the representation. We introduce two strategies to tackle this: (i) Positional Information Pre-training and (ii) Contrastive Learning on PI using Cross-Modality Matching. Doing so, the model can correctly classify if images with detailed PI statements match. Additionally to the 2D information from bounding boxes, we introduce the object&apos;s depth as new feature for a better object localization in the space. Even though we were able to improve the model properties as defined by our

Název v anglickém jazyce

Probing the Role of Positional Information in Vision-Language Models

Popis výsledku anglicky

In most Vision-Language models (VL), the understanding of the image structure is enabled by injecting the position information (PI) about objects in the image. In our case study of LXMERT, a state-of-the-art VL model, we probe the use of the PI in the representation and study its effect on Visual Question Answering. We show that the model is not capable of leveraging the PI for the image-text matching task on a challenge set where only position differs. Yet, our experiments with probing confirm that the PI is indeed present in the representation. We introduce two strategies to tackle this: (i) Positional Information Pre-training and (ii) Contrastive Learning on PI using Cross-Modality Matching. Doing so, the model can correctly classify if images with detailed PI statements match. Additionally to the 2D information from bounding boxes, we introduce the object&apos;s depth as new feature for a better object localization in the space. Even though we were able to improve the model properties as defined by our

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GX19-26934X" target="_blank" >GX19-26934X: Neuronové reprezentace v multimodálním a mnohojazyčném modelování</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2022

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Findings of the Association for Computational Linguistics: NAACL 2022

ISBN

978-1-955917-76-6

ISSN

—

e-ISSN

—

Počet stran výsledku

11

Strana od-do

1031-1041

Název nakladatele

Association for Computational Linguistics

Místo vydání

Stroudsburg, PA, USA

Místo konání akce

Seattle, WA, USA

Datum konání akce

10. 7. 2022

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—