SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A10511669" target="_blank" >RIV/00216208:11320/25:10511669 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2025.semeval-1.322" target="_blank" >https://aclanthology.org/2025.semeval-1.322</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
Original language description
We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs).Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task. We received 2,618 submissions from 43 participating teams employing diverse methodologies. The very high number of submissions highlights the interest of the community in hallucination detection. We present the results of the participating systems and provide an empirical analysis in order to better understand the factors that can lead to strong performance in this task. We also underscore current challenges, notably the varying degree of hallucinations across languages and the high annotator disagreement when labeling hallucination spans.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2025
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025)
ISBN
979-8-89176-273-2
ISSN
—
e-ISSN
—
Number of pages
26
Pages from-to
2472-2497
Publisher name
Association for Computational Lingustics
Place of publication
Kerrville, TX, USA
Event location
Wien, Austria
Event date
Jul 31, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—