Improving the Peer-Review Process with Model-Based Estimates of Inter-Rater Reliability and Detection of Rating Bias: From Teacher Selection to Journal Submissions and Grant Applications
Identifikátory výsledku
Kód výsledku v IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F18%3A00503448" target="_blank" >RIV/67985807:_____/18:00503448 - isvavai.cz</a>
Výsledek na webu
<a href="http://www.peere.org/wp-content/uploads/2018/10/PEERE2018_martinkovaIRR.pdf" target="_blank" >http://www.peere.org/wp-content/uploads/2018/10/PEERE2018_martinkovaIRR.pdf</a>
DOI - Digital Object Identifier
—
Alternativní jazyky
Jazyk výsledku
angličtina
Název v původním jazyce
Improving the Peer-Review Process with Model-Based Estimates of Inter-Rater Reliability and Detection of Rating Bias: From Teacher Selection to Journal Submissions and Grant Applications
Popis výsledku v původním jazyce
ZÁKLADNÍ ÚDAJE: PEERE Conference on Peer Review, Rome 2018: Presentations. PEERE 2018: Conference on Peer Review. 07.03.2018-09.03.2018, Rome. Grant CEP: GA ČR GJ15-15856Y. ABSTRAKT: In this work we develop and apply statistical models for analysis and advanced estimation of inter-rater reliability (IRR) and for detection of bias in peer review. We apply the models to ratings of applications to teacher positions. We show bias for ratings of applicants external to the district, which remains significant even after various measures of teacher quality are included in the model. Using model-based IRR we moreover demonstrate significantly lower inter-rater reliability for ratings of external applicants. We discuss implications for teacher hiring practices as well as possibilities for application of the proposed methodologies in journal and grant peer review.
Název v anglickém jazyce
Improving the Peer-Review Process with Model-Based Estimates of Inter-Rater Reliability and Detection of Rating Bias: From Teacher Selection to Journal Submissions and Grant Applications
Popis výsledku anglicky
ZÁKLADNÍ ÚDAJE: PEERE Conference on Peer Review, Rome 2018: Presentations. PEERE 2018: Conference on Peer Review. 07.03.2018-09.03.2018, Rome. Grant CEP: GA ČR GJ15-15856Y. ABSTRAKT: In this work we develop and apply statistical models for analysis and advanced estimation of inter-rater reliability (IRR) and for detection of bias in peer review. We apply the models to ratings of applications to teacher positions. We show bias for ratings of applicants external to the district, which remains significant even after various measures of teacher quality are included in the model. Using model-based IRR we moreover demonstrate significantly lower inter-rater reliability for ratings of external applicants. We discuss implications for teacher hiring practices as well as possibilities for application of the proposed methodologies in journal and grant peer review.
Klasifikace
Druh
O - Ostatní výsledky
CEP obor
—
OECD FORD obor
10103 - Statistics and probability
Návaznosti výsledku
Projekt
<a href="/cs/project/GJ15-15856Y" target="_blank" >GJ15-15856Y: Odhad psychometrických vlastností jako součást vývoje přijímacích testů</a><br>
Návaznosti
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Ostatní
Rok uplatnění
2018
Kód důvěrnosti údajů
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů