Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability
Identifikátory výsledku
Kód výsledku v IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11410%2F18%3A10380400" target="_blank" >RIV/00216208:11410/18:10380400 - isvavai.cz</a>
Nalezeny alternativní kódy
RIV/67985807:_____/18:00494344
Výsledek na webu
<a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0203002" target="_blank" >https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0203002</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1371/journal.pone.0203002" target="_blank" >10.1371/journal.pone.0203002</a>
Alternativní jazyky
Jazyk výsledku
angličtina
Název v původním jazyce
Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability
Popis výsledku v původním jazyce
Ratings are present in many areas of assessment including peer review of research proposals and journal articles, teacher observations, university admissions and selection of new hires. One feature present in any rating process with multiple raters is that different raters often assign different scores to the same assessee, with the potential for bias and inconsistencies related to rater or assessee covariates. This paper analyzes disparities in ratings of internal and external applicants to teaching positions using applicant data from Spokane Public Schools. We first test for biases in rating while accounting for measures of teacher applicant qualifications and quality. Then, we develop model-based inter-rater reliability (IRR) estimates that allow us to account for various sources of measurement error, the hierarchical structure of the data, and to test whether covariates, such as applicant status, moderate IRR. We find that applicants external to the district receive lower ratings for job applications compared to internal applicants. This gap in ratings remains significant even after including measures of qualifications and quality such as experience, state licensure scores, or estimated teacher value added. With model-based IRR, we further show that consistency between raters is significantly lower when rating external applicants. We conclude the paper by discussing policy implications and possible applications of our model-based IRR estimate for hiring and selection practices in and out of the teacher labor market.
Název v anglickém jazyce
Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability
Popis výsledku anglicky
Ratings are present in many areas of assessment including peer review of research proposals and journal articles, teacher observations, university admissions and selection of new hires. One feature present in any rating process with multiple raters is that different raters often assign different scores to the same assessee, with the potential for bias and inconsistencies related to rater or assessee covariates. This paper analyzes disparities in ratings of internal and external applicants to teaching positions using applicant data from Spokane Public Schools. We first test for biases in rating while accounting for measures of teacher applicant qualifications and quality. Then, we develop model-based inter-rater reliability (IRR) estimates that allow us to account for various sources of measurement error, the hierarchical structure of the data, and to test whether covariates, such as applicant status, moderate IRR. We find that applicants external to the district receive lower ratings for job applications compared to internal applicants. This gap in ratings remains significant even after including measures of qualifications and quality such as experience, state licensure scores, or estimated teacher value added. With model-based IRR, we further show that consistency between raters is significantly lower when rating external applicants. We conclude the paper by discussing policy implications and possible applications of our model-based IRR estimate for hiring and selection practices in and out of the teacher labor market.
Klasifikace
Druh
J<sub>imp</sub> - Článek v periodiku v databázi Web of Science
CEP obor
—
OECD FORD obor
50301 - Education, general; including training, pedagogy, didactics [and education systems]
Návaznosti výsledku
Projekt
<a href="/cs/project/GJ15-15856Y" target="_blank" >GJ15-15856Y: Odhad psychometrických vlastností jako součást vývoje přijímacích testů</a><br>
Návaznosti
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Ostatní
Rok uplatnění
2018
Kód důvěrnosti údajů
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Údaje specifické pro druh výsledku
Název periodika
PLoS One
ISSN
1932-6203
e-ISSN
—
Svazek periodika
13
Číslo periodika v rámci svazku
10
Stát vydavatele periodika
US - Spojené státy americké
Počet stran výsledku
17
Strana od-do
1-17
Kód UT WoS článku
000446545500004
EID výsledku v databázi Scopus
2-s2.0-85054464213