Why rankings of biomedical image analysis competitions should be interpreted with care

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14330%2F18%3A00101338" target="_blank" >RIV/00216224:14330/18:00101338 - isvavai.cz</a>

Výsledek na webu

<a href="http://doi.org/10.1038/s41467-018-07619-7" target="_blank" >http://doi.org/10.1038/s41467-018-07619-7</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1038/s41467-018-07619-7" target="_blank" >10.1038/s41467-018-07619-7</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Why rankings of biomedical image analysis competitions should be interpreted with care

Popis výsledku v původním jazyce

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Název v anglickém jazyce

Why rankings of biomedical image analysis competitions should be interpreted with care

Popis výsledku anglicky

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2018

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Nature Communications

ISSN

2041-1723

e-ISSN

—

Svazek periodika

9

Číslo periodika v rámci svazku

5217

Stát vydavatele periodika

CH - Švýcarská konfederace

Počet stran výsledku

13

Strana od-do

1-13

Kód UT WoS článku

000452282700012

EID výsledku v databázi Scopus

—