On Model Evaluation Under Non-constant Class Imbalance
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F20%3A00342402" target="_blank" >RIV/68407700:21230/20:00342402 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1007/978-3-030-50423-6_6" target="_blank" >https://doi.org/10.1007/978-3-030-50423-6_6</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/978-3-030-50423-6_6" target="_blank" >10.1007/978-3-030-50423-6_6</a>
Alternative languages
Result language
angličtina
Original language name
On Model Evaluation Under Non-constant Class Imbalance
Original language description
Many real-world classification problems are significantly class-imbalanced to detriment of the class of interest. The standard set of proper evaluation metrics is well-known but the usual assumption is that the test dataset imbalance equals the real-world imbalance. In practice, this assumption is often broken for various reasons. The reported results are then often too optimistic and may lead to wrong conclusions about industrial impact and suitability of proposed techniques. We introduce methods (Supplementary code related to techniques described in this paper is available at: https://github.com/CiscoCTA/nci_eval) focusing on evaluation under non-constant class imbalance. We show that not only the absolute values of commonly used metrics, but even the order of classifiers in relation to the evaluation metric used is affected by the change of the imbalance rate. Finally, we demonstrate that using subsampling in order to get a test dataset with class imbalance equal to the one observed in the wild is not necessary, and eventually can lead to significant errors in classifier’s performance estimate.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Research Center for Informatics</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2020
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Computational Science - ICCS 2020
ISBN
978-3-030-50422-9
ISSN
0302-9743
e-ISSN
1611-3349
Number of pages
14
Pages from-to
74-87
Publisher name
Springer
Place of publication
Cham
Event location
Amsterdam
Event date
Jun 3, 2020
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—