Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F23%3A00580447" target="_blank" >RIV/67985807:_____/23:00580447 - isvavai.cz</a>

Result on the web

<a href="https://eam2023.ugent.be/images/eam2023_abstracts_book.pdf" target="_blank" >https://eam2023.ugent.be/images/eam2023_abstracts_book.pdf</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance

Original language description

ZÁKLADNÍ ÚDAJE: The 10th European Congress of Methodology (EAM2023) Book of Abstracts. Ghent: Ghent University, 2023. s. 26-26. [EAM2023: European Congress of Methodology /10./. 11.07.2023-13.07.2023, Ghent]. ABSTRAKT: Various properties of text wording of a given test item determine how difficult the item is for a test-taker. While the item difficulty is commonly estimated using item response theory (IRT) models based on test-takers’ responses, information on item difficulty is encoded in its text and could be predicted using machine-learning algorithms. In this work, we used text wordings of test items of the reading comprehension part of a test of English as a foreign language. For each item, we tokenized and lemmatized item text, removed stopwords, and calculated various features such as word counts, readability indices, lexical frequencies, and measures of item parts’ similarity. Then, the resulting dataset containing text features in rows was enriched by item difficulty estimated using the Rasch model. The item difficulty was predicted using multiple machine-learning supervised algorithms of regression task. Firstly, we applied regularization algorithms, i.e., LASSO, ridge regression, and elastic net, to select appropriate features, reduce dimensionality, and predict the (continuous) difficulty. Besides that, we employed support vector machines, regression trees and forests, and neural networks. Once we categorized the difficulty into disjunctive intervals, we switched the regression into a classification task, also applying the naïve Bayes classifier. To compare the algorithms to each other and domain experts’ difficulty predictions, we learned algorithms many times within cross-validation and estimated root mean square errors and predictive accuracies for each approach. Regularization algorithms in regression tasks and random forests in classification seemed to outperform other algorithms and predicted item difficulty similarly to domain experts

Czech name

—

Czech description

—

Type

O - Miscellaneous

CEP classification

—

OECD FORD branch

50301 - Education, general; including training, pedagogy, didactics [and education systems]

Project

Result was created during the realization of more than one project. More information in the Projects tab.

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů