A Domain-region Based Evaluation of ML Performance Robustness to Covariate Shift
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F23%3A00370203" target="_blank" >RIV/68407700:21230/23:00370203 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1007/s00521-023-08622-w" target="_blank" >https://doi.org/10.1007/s00521-023-08622-w</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/s00521-023-08622-w" target="_blank" >10.1007/s00521-023-08622-w</a>
Alternative languages
Result language
angličtina
Original language name
A Domain-region Based Evaluation of ML Performance Robustness to Covariate Shift
Original language description
Most machine learning methods assume that the input data distribution is the same in the training and testing phases. However, in practice, this stationarity is usually not met and the distribution of inputs differs, leading to unexpected performance of the learned model in deployment. The issue in which the training and test data inputs follow different probability distributions while the input-output relationship remains unchanged is referred to as covariate shift. In this paper, the performance of conventional machine learning models was experimentally evaluated in the presence of covariate shift. Furthermore, a region-based evaluation was performed by decomposing the domain of probability density function of the input data to assess the classifier's performance per domain region. Distributional changes were simulated in a two-dimensional classification problem. Subsequently, a higher four-dimensional experiments were conducted. Based on the experimental analysis, the Random Forests algorithm is the most robust classifier in the two-dimensional case, showing the lowest degradation rate for accuracy and F1-score metrics, with a range between 0.1% and 2.08%. Moreover, the results reveal that in higher-dimensional experiments, the performance of the models is predominantly influenced by the complexity of the classification function, leading to degradation rates exceeding 25% in most cases. It is also concluded that the models exhibit high bias toward the region with high density in the input space domain of the training samples.
Czech name
—
Czech description
—
Classification
Type
J<sub>imp</sub> - Article in a specialist periodical, which is included in the Web of Science database
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2023
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Name of the periodical
Neural Computing and Applications
ISSN
0941-0643
e-ISSN
1433-3058
Volume of the periodical
35
Issue of the periodical within the volume
24
Country of publishing house
GB - UNITED KINGDOM
Number of pages
23
Pages from-to
17555-17577
UT code for WoS article
000988193400003
EID of the result in the Scopus database
2-s2.0-85159332578