When Zero May not be Zero: A Cautionary Note on the use of Inter-rater Reliability in Evaluating Grant Peer Review
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F21%3A00541889" target="_blank" >RIV/67985807:_____/21:00541889 - isvavai.cz</a>
Result on the web
<a href="http://dx.doi.org/10.1111/rssa.12681" target="_blank" >http://dx.doi.org/10.1111/rssa.12681</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1111/rssa.12681" target="_blank" >10.1111/rssa.12681</a>
Alternative languages
Result language
angličtina
Original language name
When Zero May not be Zero: A Cautionary Note on the use of Inter-rater Reliability in Evaluating Grant Peer Review
Original language description
Considerable attention has focused on studying reviewer agreement via inter‐rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported an IRR of zero in the mock peer review of top‐quality grant proposals, we use real data from a complete range of submissions to the National Institutes of Health and to the American Institute of Biological Sciences to bring awareness to two important issues with using IRR for assessing peer review quality. First, we demonstrate that estimating local IRR from subsets of restricted‐quality proposals will likely result in zero estimates under many scenarios. In both data sets, we find that zero local IRR estimates are more likely when subsets of top‐quality proposals rather than bottom‐quality proposals are considered. However, zero estimates from range‐restricted data should not be interpreted as indicating arbitrariness in peer review. On the contrary, despite different scoring scales used by the two agencies, when complete ranges of proposals are considered, IRR estimates are above 0.6 which indicates good reviewer agreement. Furthermore, we demonstrate that, with a small number of reviewers per proposal, zero estimates of IRR are possible even when the true value is not zero.
Czech name
—
Czech description
—
Classification
Type
J<sub>imp</sub> - Article in a specialist periodical, which is included in the Web of Science database
CEP classification
—
OECD FORD branch
10103 - Statistics and probability
Result continuities
Project
<a href="/en/project/GA21-03658S" target="_blank" >GA21-03658S: Theoretical foundations of computational psychometrics</a><br>
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Name of the periodical
Journal of the Royal Statistical Society Series A-Statistics in Society
ISSN
0964-1998
e-ISSN
1467-985X
Volume of the periodical
184
Issue of the periodical within the volume
3
Country of publishing house
US - UNITED STATES
Number of pages
16
Pages from-to
904-919
UT code for WoS article
000641375900001
EID of the result in the Scopus database
2-s2.0-85104530726