Can Large Language Models Understand Context?

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AG69F4JPZ" target="_blank" >RIV/00216208:11320/25:G69F4JPZ - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85188697856&partnerID=40&md5=e60959749589979204fe03e35147f6b9" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85188697856&partnerID=40&md5=e60959749589979204fe03e35147f6b9</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Can Large Language Models Understand Context?

Original language description

Understanding context is key to understanding human language, an ability which Large Language Models (LLMs) have been increasingly seen to demonstrate to an impressive extent. However, though the evaluation of LLMs encompasses various domains within the realm of Natural Language Processing, limited attention has been paid to probing their linguistic capability of understanding contextual features. This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. This benchmark comprises of four distinct tasks and nine datasets, all featuring prompts designed to assess the models’ ability to understand context. First, we evaluate the performance of LLMs under the in-context learning pretraining scenario. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. Second, as LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings. We find that 3-bit post-training quantization leads to varying degrees of performance reduction on our benchmark. We conduct an extensive analysis of these scenarios to substantiate our experimental results. © 2024 Association for Computational Linguistics.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

EACL - Conf. Eur. Chapter Assoc. Comput. Linguist., Find. EACL

ISBN

979-889176093-6

ISSN

—

e-ISSN

—

Number of pages

15

Pages from-to

2004-2018

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

—

Event location

St. Julian's

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—