In the eyes of a language model: A comprehensive examination through eye-tracking data

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AWFLHUX98" target="_blank" >RIV/00216208:11320/26:WFLHUX98 - isvavai.cz</a>

Result on the web

<a href="http://dx.doi.org/10.1016/j.neucom.2025.130617" target="_blank" >http://dx.doi.org/10.1016/j.neucom.2025.130617</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.neucom.2025.130617" target="_blank" >10.1016/j.neucom.2025.130617</a>

Result language

angličtina

Original language name

In the eyes of a language model: A comprehensive examination through eye-tracking data

Original language description

Cognitive signals, particularly eye-tracking data, offer a unique lens for understanding human sentence processing. Leveraging eye-gaze data from the English and Italian section of the Multilingual Eye-Movement Corpus (MECO), we designed a series of experiments aiming at exploring whether pre-trained neural language models (NLMs) encode patterns representative of human reading behavior and if directly incorporating this information through a fine-tuning process influences the cognitive plausibility of the model. Additionally, we sought to determine if such an impact persists through a downstream task. Our findings reveal that transformers encode eye-gaze-related information during pretraining and that explicitly integrating eye-tracking features increases model alignment with human attention. When investigating the effect of intermediate fine-tuning on eye-tracking data on the model's performance on a downstream task, we observe that this intermediate step does not result in catastrophic forgetting, despite the very different nature of the considered downstream task. In addition, the attention mechanism of models undergoing intermediate fine-tuning remains closely aligned with human attention. In conclusion, our comprehensive evaluation of NLMs informed by human attention patterns offers great potential for advancing the growing field of eXplainable Artificial Intelligence (XAI). Grounding language models in real-world cognitive processes enables the creation of systems that not only replicate human language output but also align with the cognitive mechanisms behind reading and comprehension. This alignment with human behavior enhances model adaptability, interpretability, and effectiveness, fostering more human-centric, transparent, and reliable AI applications across various domains.1 © 2025 The Authors

Czech name

—

Czech description

—

Type

J_SC - Article in a specialist periodical, which is included in the SCOPUS database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Neurocomputing

ISSN

0925-2312

e-ISSN

—

Volume of the periodical

650

Issue of the periodical within the volume

2025

Country of publishing house

US - UNITED STATES

Number of pages

27

Pages from-to

130617

UT code for WoS article

—

EID of the result in the Scopus database

2-s2.0-105010133396