Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AX3CS2S63" target="_blank" >RIV/00216208:11320/26:X3CS2S63 - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2025.acl-long.362/" target="_blank" >https://aclanthology.org/2025.acl-long.362/</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2025.acl-long.362" target="_blank" >10.18653/v1/2025.acl-long.362</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context

Popis výsledku v původním jazyce

Human processing of idioms heavily depends on interpreting the surrounding context in which they appear. While large language models (LLMs) have achieved impressive performance on idiomaticity detection benchmarks, this success may be driven by reasoning shortcuts present in existing datasets. To address this, we introduce a novel, controlled contrastive dataset (DICE) specifically designed to assess whether LLMs can effectively leverage context to disambiguate idiomatic meanings. Furthermore, we investigate the influence of collocational frequency and sentence probability—proxies for human processing known to affect idiom resolution—on model performance. Our results show that LLMs frequently fail to resolve idiomaticity when it depends on contextual understanding, performing better on sentences deemed more likely by the model. Additionally, idiom frequency influences performance but does not guarantee accurate interpretation. Our findings emphasize the limitations of current models in grasping contextual meaning and highlight the need for more context-sensitive evaluation.

Název v anglickém jazyce

Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context

Popis výsledku anglicky

Human processing of idioms heavily depends on interpreting the surrounding context in which they appear. While large language models (LLMs) have achieved impressive performance on idiomaticity detection benchmarks, this success may be driven by reasoning shortcuts present in existing datasets. To address this, we introduce a novel, controlled contrastive dataset (DICE) specifically designed to assess whether LLMs can effectively leverage context to disambiguate idiomatic meanings. Furthermore, we investigate the influence of collocational frequency and sentence probability—proxies for human processing known to affect idiom resolution—on model performance. Our results show that LLMs frequently fail to resolve idiomaticity when it depends on contextual understanding, performing better on sentences deemed more likely by the model. Additionally, idiom frequency influences performance but does not guarantee accurate interpretation. Our findings emphasize the limitations of current models in grasping contextual meaning and highlight the need for more context-sensitive evaluation.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

ISBN

979-8-89176-251-0

ISSN

—

e-ISSN

—

Počet stran výsledku

19

Strana od-do

7314-7332

Název nakladatele

Association for Computational Linguistics

Místo vydání

—

Místo konání akce

Vienna, Austria

Datum konání akce

1. 1. 2026

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—