Probing of pretrained multilingual models on the knowledge of discourse
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AE2XLQPL8" target="_blank" >RIV/00216208:11320/25:E2XLQPL8 - isvavai.cz</a>
Result on the web
<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85190284099&partnerID=40&md5=de6802839bb7ec70229f4e02cc2afff0" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85190284099&partnerID=40&md5=de6802839bb7ec70229f4e02cc2afff0</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
Probing of pretrained multilingual models on the knowledge of discourse
Original language description
With the raise of large language models (LLMs), different evaluation methods, including probing methods, are gaining more attention. Probing methods are meant to evaluate LLMs on their linguistic abilities. However, most of the studies are focused on morphology and syntax, leaving discourse research out of the scope. At the same time, understanding discourse and pragmatics is crucial to building up the conversational abilities of models. In this paper, we address the problem of probing several models of discourse knowledge in 10 languages. We present an algorithm to automatically adapt existing discourse tasks to other languages based on the Universal Dependencies (UD) annotation. We find that models perform similarly on high- and low-resourced languages. However, the overall low performance of the models’ quality shows that they do not acquire discourse well enough. © 2024 Association for Computational Linguistics.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
CODI - Workshop Comput. Approaches Discourse, Proc. Workshop
ISBN
979-889176079-0
ISSN
—
e-ISSN
—
Number of pages
13
Pages from-to
78-90
Publisher name
Association for Computational Linguistics (ACL)
Place of publication
—
Event location
St. Julian's, Malta
Event date
Jan 1, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—