Improving the Peer-Review Process with Model-Based Estimates of Inter-Rater Reliability and Detection of Rating Bias: From Teacher Selection to Journal Submissions and Grant Applications

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F18%3A00503448" target="_blank" >RIV/67985807:_____/18:00503448 - isvavai.cz</a>
Result on the web
<a href="http://www.peere.org/wp-content/uploads/2018/10/PEERE2018_martinkovaIRR.pdf" target="_blank" >http://www.peere.org/wp-content/uploads/2018/10/PEERE2018_martinkovaIRR.pdf</a>
DOI - Digital Object Identifier
—

Result language
angličtina
Original language name
Improving the Peer-Review Process with Model-Based Estimates of Inter-Rater Reliability and Detection of Rating Bias: From Teacher Selection to Journal Submissions and Grant Applications
Original language description
ZÁKLADNÍ ÚDAJE: PEERE Conference on Peer Review, Rome 2018: Presentations. PEERE 2018: Conference on Peer Review. 07.03.2018-09.03.2018, Rome. Grant CEP: GA ČR GJ15-15856Y. ABSTRAKT: In this work we develop and apply statistical models for analysis and advanced estimation of inter-rater reliability (IRR) and for detection of bias in peer review. We apply the models to ratings of applications to teacher positions. We show bias for ratings of applicants external to the district, which remains significant even after various measures of teacher quality are included in the model. Using model-based IRR we moreover demonstrate significantly lower inter-rater reliability for ratings of external applicants. We discuss implications for teacher hiring practices as well as possibilities for application of the proposed methodologies in journal and grant peer review.
Czech name
—
Czech description
—

Project
<a href="/en/project/GJ15-15856Y" target="_blank" >GJ15-15856Y: Estimation of psychometric measures as part of admission test development</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year
2018
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Similar results(10)