Applying large language models for automated essay scoring for non-native Japanese

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A3AQKM2AE" target="_blank" >RIV/00216208:11320/25:3AQKM2AE - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195378013&doi=10.1057%2fs41599-024-03209-9&partnerID=40&md5=2df4ec39be1d1b6ad1fd8d241c682779" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195378013&doi=10.1057%2fs41599-024-03209-9&partnerID=40&md5=2df4ec39be1d1b6ad1fd8d241c682779</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1057/s41599-024-03209-9" target="_blank" >10.1057/s41599-024-03209-9</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Applying large language models for automated essay scoring for non-native Japanese

Popis výsledku v původním jazyce

Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs. © The Author(s) 2024.

Název v anglickém jazyce

Applying large language models for automated essay scoring for non-native Japanese

Popis výsledku anglicky

Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs. © The Author(s) 2024.

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Humanities and Social Sciences Communications

ISSN

2662-9992

e-ISSN

—

Svazek periodika

11

Číslo periodika v rámci svazku

1

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

16

Strana od-do

1-16

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-85195378013