Inter-Rater Reliability In Complex Situations: Model Selection

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F21%3A00570043" target="_blank" >RIV/67985807:_____/21:00570043 - isvavai.cz</a>

Result on the web

—

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Inter-Rater Reliability In Complex Situations: Model Selection

Original language description

ZÁKLADNÍ ÚDAJE: Inter-Rater Reliability In Complex Situations: Model Selection. [IMPS 2021: The Annual Meeting of the Psychometric Society. Virtual, 20.07.2021-23.07.2021]. ABSTRAKT: Inter-rater reliability (IRR) is a prerequisite of high-quality ratings and assessments. However, the estimates of IRR may be affected by contextual factors such as rater’s or ratee’s age, gender, major, or internal vs. external status (Martinková, Goldhaber & Erosheva, 2018). In a previous simulation study, we explored how different methods can estimate heterogeneity in IRR if the data-generating model is known (Bartoš, Martinková & Brabec, 2019). We extend the previous work by considering cases when the true data-generating model is unknown. First, we evaluate several frequentist (forward, backward, AIC, and BIC) and Bayesian (Bayes factors, LOO, and WAIC) model selection techniques in their ability to choose the true model. Second, we compare estimates of variance components and IRR for models resulting from different model selection techniques in terms of bias and RMSE. Finally, we consider model-averaged estimates which incorporate uncertainty in the model selection process into the final estimate. Our results suggest that despite the differences in the performance of model selection techniques, the model-averaged estimates perform better than the estimates based on the selected models, regardless of the model selection criteria. We conclude with discussion of further computational aspects of IRR estimation (Erosheva, Martinkova, & Lee, 2021) and of generalizations to more complex designs.nnn

Czech name

—

Czech description

—

Type

O - Miscellaneous

CEP classification

—

OECD FORD branch

10103 - Statistics and probability

Project

<a href="/en/project/GA21-03658S" target="_blank" >GA21-03658S: Theoretical foundations of computational psychometrics</a><br>

Continuities

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2021

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů