Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F23%3A00580447" target="_blank" >RIV/67985807:_____/23:00580447 - isvavai.cz</a>

Výsledek na webu

<a href="https://eam2023.ugent.be/images/eam2023_abstracts_book.pdf" target="_blank" >https://eam2023.ugent.be/images/eam2023_abstracts_book.pdf</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance

Popis výsledku v původním jazyce

ZÁKLADNÍ ÚDAJE: The 10th European Congress of Methodology (EAM2023) Book of Abstracts. Ghent: Ghent University, 2023. s. 26-26. [EAM2023: European Congress of Methodology /10./. 11.07.2023-13.07.2023, Ghent]. ABSTRAKT: Various properties of text wording of a given test item determine how difficult the item is for a test-taker. While the item difficulty is commonly estimated using item response theory (IRT) models based on test-takers’ responses, information on item difficulty is encoded in its text and could be predicted using machine-learning algorithms. In this work, we used text wordings of test items of the reading comprehension part of a test of English as a foreign language. For each item, we tokenized and lemmatized item text, removed stopwords, and calculated various features such as word counts, readability indices, lexical frequencies, and measures of item parts’ similarity. Then, the resulting dataset containing text features in rows was enriched by item difficulty estimated using the Rasch model. The item difficulty was predicted using multiple machine-learning supervised algorithms of regression task. Firstly, we applied regularization algorithms, i.e., LASSO, ridge regression, and elastic net, to select appropriate features, reduce dimensionality, and predict the (continuous) difficulty. Besides that, we employed support vector machines, regression trees and forests, and neural networks. Once we categorized the difficulty into disjunctive intervals, we switched the regression into a classification task, also applying the naïve Bayes classifier. To compare the algorithms to each other and domain experts’ difficulty predictions, we learned algorithms many times within cross-validation and estimated root mean square errors and predictive accuracies for each approach. Regularization algorithms in regression tasks and random forests in classification seemed to outperform other algorithms and predicted item difficulty similarly to domain experts

Název v anglickém jazyce

Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance

Popis výsledku anglicky

ZÁKLADNÍ ÚDAJE: The 10th European Congress of Methodology (EAM2023) Book of Abstracts. Ghent: Ghent University, 2023. s. 26-26. [EAM2023: European Congress of Methodology /10./. 11.07.2023-13.07.2023, Ghent]. ABSTRAKT: Various properties of text wording of a given test item determine how difficult the item is for a test-taker. While the item difficulty is commonly estimated using item response theory (IRT) models based on test-takers’ responses, information on item difficulty is encoded in its text and could be predicted using machine-learning algorithms. In this work, we used text wordings of test items of the reading comprehension part of a test of English as a foreign language. For each item, we tokenized and lemmatized item text, removed stopwords, and calculated various features such as word counts, readability indices, lexical frequencies, and measures of item parts’ similarity. Then, the resulting dataset containing text features in rows was enriched by item difficulty estimated using the Rasch model. The item difficulty was predicted using multiple machine-learning supervised algorithms of regression task. Firstly, we applied regularization algorithms, i.e., LASSO, ridge regression, and elastic net, to select appropriate features, reduce dimensionality, and predict the (continuous) difficulty. Besides that, we employed support vector machines, regression trees and forests, and neural networks. Once we categorized the difficulty into disjunctive intervals, we switched the regression into a classification task, also applying the naïve Bayes classifier. To compare the algorithms to each other and domain experts’ difficulty predictions, we learned algorithms many times within cross-validation and estimated root mean square errors and predictive accuracies for each approach. Regularization algorithms in regression tasks and random forests in classification seemed to outperform other algorithms and predicted item difficulty similarly to domain experts

Druh

O - Ostatní výsledky

CEP obor

—

OECD FORD obor

50301 - Education, general; including training, pedagogy, didactics [and education systems]

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů