Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AYMRTZ8PA" target="_blank" >RIV/00216208:11320/26:YMRTZ8PA - isvavai.cz</a>

Result on the web

<a href="https://aclanthology.org/2025.gem-1.41/" target="_blank" >https://aclanthology.org/2025.gem-1.41/</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.48550/arXiv.2504.09714" target="_blank" >10.48550/arXiv.2504.09714</a>

Result language

angličtina

Original language name

Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish

Original language description

The reliance on translated or adapted datasets from English or multilingual resources introduces challenges regarding linguistic and cultural suitability. This study addresses the need for robust and culturally appropriate benchmarks by evaluating the quality of 17 commonly used Turkish benchmark datasets. Using a comprehensive framework that assesses six criteria, both human and LLM-judge annotators provide detailed evaluations to identify dataset strengths and shortcomings. Our results reveal that 70% of the benchmark datasets fail to meet our heuristic quality standards. The correctness of the usage of technical terms is the strongest criterion, but 85% of the criteria are not satisfied in the examined datasets. Although LLM judges demonstrate potential, they are less effective than human annotators, particularly in understanding cultural common sense knowledge and interpreting fluent, unambiguous text. GPT-4o has stronger labeling capabilities for grammatical and technical tasks, while Llama3.3-70B excels at correctness and cultural knowledge evaluation. Our findings emphasize the urgent need for more rigorous quality control in creating and adapting datasets for low-resource languages.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM^2)

ISBN

979-8-89176-261-9

ISSN

—

e-ISSN

—

Number of pages

17

Pages from-to

471-487

Publisher name

—

Place of publication

—

Event location

Vienna, Austria

Event date

Jan 1, 2026

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—