Inter-Rater Reliability In Complex Situations: Model Selection

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F21%3A00570043" target="_blank" >RIV/67985807:_____/21:00570043 - isvavai.cz</a>

Výsledek na webu

—

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Inter-Rater Reliability In Complex Situations: Model Selection

Popis výsledku v původním jazyce

ZÁKLADNÍ ÚDAJE: Inter-Rater Reliability In Complex Situations: Model Selection. [IMPS 2021: The Annual Meeting of the Psychometric Society. Virtual, 20.07.2021-23.07.2021]. ABSTRAKT: Inter-rater reliability (IRR) is a prerequisite of high-quality ratings and assessments. However, the estimates of IRR may be affected by contextual factors such as rater’s or ratee’s age, gender, major, or internal vs. external status (Martinková, Goldhaber & Erosheva, 2018). In a previous simulation study, we explored how different methods can estimate heterogeneity in IRR if the data-generating model is known (Bartoš, Martinková & Brabec, 2019). We extend the previous work by considering cases when the true data-generating model is unknown. First, we evaluate several frequentist (forward, backward, AIC, and BIC) and Bayesian (Bayes factors, LOO, and WAIC) model selection techniques in their ability to choose the true model. Second, we compare estimates of variance components and IRR for models resulting from different model selection techniques in terms of bias and RMSE. Finally, we consider model-averaged estimates which incorporate uncertainty in the model selection process into the final estimate. Our results suggest that despite the differences in the performance of model selection techniques, the model-averaged estimates perform better than the estimates based on the selected models, regardless of the model selection criteria. We conclude with discussion of further computational aspects of IRR estimation (Erosheva, Martinkova, & Lee, 2021) and of generalizations to more complex designs.nnn

Název v anglickém jazyce

Inter-Rater Reliability In Complex Situations: Model Selection

Popis výsledku anglicky

ZÁKLADNÍ ÚDAJE: Inter-Rater Reliability In Complex Situations: Model Selection. [IMPS 2021: The Annual Meeting of the Psychometric Society. Virtual, 20.07.2021-23.07.2021]. ABSTRAKT: Inter-rater reliability (IRR) is a prerequisite of high-quality ratings and assessments. However, the estimates of IRR may be affected by contextual factors such as rater’s or ratee’s age, gender, major, or internal vs. external status (Martinková, Goldhaber & Erosheva, 2018). In a previous simulation study, we explored how different methods can estimate heterogeneity in IRR if the data-generating model is known (Bartoš, Martinková & Brabec, 2019). We extend the previous work by considering cases when the true data-generating model is unknown. First, we evaluate several frequentist (forward, backward, AIC, and BIC) and Bayesian (Bayes factors, LOO, and WAIC) model selection techniques in their ability to choose the true model. Second, we compare estimates of variance components and IRR for models resulting from different model selection techniques in terms of bias and RMSE. Finally, we consider model-averaged estimates which incorporate uncertainty in the model selection process into the final estimate. Our results suggest that despite the differences in the performance of model selection techniques, the model-averaged estimates perform better than the estimates based on the selected models, regardless of the model selection criteria. We conclude with discussion of further computational aspects of IRR estimation (Erosheva, Martinkova, & Lee, 2021) and of generalizations to more complex designs.nnn

Druh

O - Ostatní výsledky

CEP obor

—

OECD FORD obor

10103 - Statistics and probability

Projekt

<a href="/cs/project/GA21-03658S" target="_blank" >GA21-03658S: Teoretické základy výpočetní psychometrie</a><br>

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů