Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AYMRTZ8PA" target="_blank" >RIV/00216208:11320/26:YMRTZ8PA - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2025.gem-1.41/" target="_blank" >https://aclanthology.org/2025.gem-1.41/</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.48550/arXiv.2504.09714" target="_blank" >10.48550/arXiv.2504.09714</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish

Popis výsledku v původním jazyce

The reliance on translated or adapted datasets from English or multilingual resources introduces challenges regarding linguistic and cultural suitability. This study addresses the need for robust and culturally appropriate benchmarks by evaluating the quality of 17 commonly used Turkish benchmark datasets. Using a comprehensive framework that assesses six criteria, both human and LLM-judge annotators provide detailed evaluations to identify dataset strengths and shortcomings. Our results reveal that 70% of the benchmark datasets fail to meet our heuristic quality standards. The correctness of the usage of technical terms is the strongest criterion, but 85% of the criteria are not satisfied in the examined datasets. Although LLM judges demonstrate potential, they are less effective than human annotators, particularly in understanding cultural common sense knowledge and interpreting fluent, unambiguous text. GPT-4o has stronger labeling capabilities for grammatical and technical tasks, while Llama3.3-70B excels at correctness and cultural knowledge evaluation. Our findings emphasize the urgent need for more rigorous quality control in creating and adapting datasets for low-resource languages.

Název v anglickém jazyce

Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish

Popis výsledku anglicky

The reliance on translated or adapted datasets from English or multilingual resources introduces challenges regarding linguistic and cultural suitability. This study addresses the need for robust and culturally appropriate benchmarks by evaluating the quality of 17 commonly used Turkish benchmark datasets. Using a comprehensive framework that assesses six criteria, both human and LLM-judge annotators provide detailed evaluations to identify dataset strengths and shortcomings. Our results reveal that 70% of the benchmark datasets fail to meet our heuristic quality standards. The correctness of the usage of technical terms is the strongest criterion, but 85% of the criteria are not satisfied in the examined datasets. Although LLM judges demonstrate potential, they are less effective than human annotators, particularly in understanding cultural common sense knowledge and interpreting fluent, unambiguous text. GPT-4o has stronger labeling capabilities for grammatical and technical tasks, while Llama3.3-70B excels at correctness and cultural knowledge evaluation. Our findings emphasize the urgent need for more rigorous quality control in creating and adapting datasets for low-resource languages.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM^2)

ISBN

979-8-89176-261-9

ISSN

—

e-ISSN

—

Počet stran výsledku

17

Strana od-do

471-487

Název nakladatele

—

Místo vydání

—

Místo konání akce

Vienna, Austria

Datum konání akce

1. 1. 2026

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—