Eye-movement-prompted large image captioning model

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AN6BWVMQQ" target="_blank" >RIV/00216208:11320/26:N6BWVMQQ - isvavai.cz</a>

Result on the web

<a href="http://dx.doi.org/10.1016/j.patcog.2024.111097" target="_blank" >http://dx.doi.org/10.1016/j.patcog.2024.111097</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.patcog.2024.111097" target="_blank" >10.1016/j.patcog.2024.111097</a>

Result language

angličtina

Original language name

Eye-movement-prompted large image captioning model

Original language description

Pretrained large vision-language models have shown outstanding performance on the task of image captioning. However, owing to the insufficient decoding of image features, existing large models sometimes lose important information, such as objects, scenes, and their relationships. In addition, the complex “black-box” nature of these models makes their mechanisms difficult to explain. Research shows that humans learn richer representations than machines do, which inspires us to improve the accuracy and interpretability of large image captioning models by combining human observation patterns. We built a new dataset, called saliency in image captioning (SIC), to explore relationships between human vision and language representation. One thousand images with rich context information were selected as image data of SIC. Each image was annotated with five caption labels and five eye-movement labels. Through analysis of the eye-movement data, we found that humans efficiently captured comprehensive information for image captioning during their observations. Therefore, we propose an eye-movement-prompted large image captioning model, which is embedded with two carefully designed modules: the eye-movement simulation module (EMS) and the eye-movement analyzing module (EMA). EMS combines the human observation pattern to simulate eye-movement features, including the positions and scan paths of eye fixations. EMA is a graph neural network (GNN) based module, which decodes graphical eye-movement data and abstracts image features as a directed graph. More accurate descriptions can be predicted by decoding the generated graph. Extensive experiments were conducted on the MS-COCO and NoCaps datasets to validate our model. The experimental results showed that our network was interpretable, and could achieve superior results compared with state-of-the-art methods, i.e., 84.2% BLEU-4 and 145.1% CIDEr-D on MS-COCO Karpathy test split, indicating its strong potential for use in image captioning. © 2024

Czech name

—

Czech description

—

Type

J_SC - Article in a specialist periodical, which is included in the SCOPUS database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Pattern Recognition

ISSN

0031-3203

e-ISSN

—

Volume of the periodical

159

Issue of the periodical within the volume

2025

Country of publishing house

US - UNITED STATES

Number of pages

13

Pages from-to

1-13

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-85208384891