Probing of pretrained multilingual models on the knowledge of discourse

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AE2XLQPL8" target="_blank" >RIV/00216208:11320/25:E2XLQPL8 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85190284099&partnerID=40&md5=de6802839bb7ec70229f4e02cc2afff0" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85190284099&partnerID=40&md5=de6802839bb7ec70229f4e02cc2afff0</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Probing of pretrained multilingual models on the knowledge of discourse

Popis výsledku v původním jazyce

With the raise of large language models (LLMs), different evaluation methods, including probing methods, are gaining more attention. Probing methods are meant to evaluate LLMs on their linguistic abilities. However, most of the studies are focused on morphology and syntax, leaving discourse research out of the scope. At the same time, understanding discourse and pragmatics is crucial to building up the conversational abilities of models. In this paper, we address the problem of probing several models of discourse knowledge in 10 languages. We present an algorithm to automatically adapt existing discourse tasks to other languages based on the Universal Dependencies (UD) annotation. We find that models perform similarly on high- and low-resourced languages. However, the overall low performance of the models’ quality shows that they do not acquire discourse well enough. © 2024 Association for Computational Linguistics.

Název v anglickém jazyce

Probing of pretrained multilingual models on the knowledge of discourse

Popis výsledku anglicky

With the raise of large language models (LLMs), different evaluation methods, including probing methods, are gaining more attention. Probing methods are meant to evaluate LLMs on their linguistic abilities. However, most of the studies are focused on morphology and syntax, leaving discourse research out of the scope. At the same time, understanding discourse and pragmatics is crucial to building up the conversational abilities of models. In this paper, we address the problem of probing several models of discourse knowledge in 10 languages. We present an algorithm to automatically adapt existing discourse tasks to other languages based on the Universal Dependencies (UD) annotation. We find that models perform similarly on high- and low-resourced languages. However, the overall low performance of the models’ quality shows that they do not acquire discourse well enough. © 2024 Association for Computational Linguistics.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

CODI - Workshop Comput. Approaches Discourse, Proc. Workshop

ISBN

979-889176079-0

ISSN

—

e-ISSN

—

Počet stran výsledku

13

Strana od-do

78-90

Název nakladatele

Association for Computational Linguistics (ACL)

Místo vydání

—

Místo konání akce

St. Julian's, Malta

Datum konání akce

1. 1. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—