Binary cross-entropy with dynamical clipping

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61988987%3A17610%2F22%3AA2302717" target="_blank" >RIV/61988987:17610/22:A2302717 - isvavai.cz</a>

Result on the web

<a href="https://link.springer.com/article/10.1007/s00521-022-07091-x" target="_blank" >https://link.springer.com/article/10.1007/s00521-022-07091-x</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s00521-022-07091-x" target="_blank" >10.1007/s00521-022-07091-x</a>

Result language

angličtina

Original language name

Binary cross-entropy with dynamical clipping

Original language description

We investigate the adverse effect of noisy labels in a training dataset on a neural network's precision in an image classification task. The importance of this research lies in the fact that most datasets include noisy labels. To reduce the impact of noisy labels, we propose to extend the binary cross-entropy by dynamical clipping, which clips all samples' loss values in a mini-batch by a clipping constant. Such a constant is dynamically determined for every single mini-batch using its statistics. The advantage is the dynamic adaptation to any number of noisy labels in a training dataset. Thanks to that, the proposed binary cross-entropy with dynamical clipping can be used in any model utilizing cross-entropy or focal loss, including pre-trained models. We prove that the proposed loss function is an alpha-calibrated classification loss, implying consistency and robustness to noise misclassification in more general asymmetric problems. We demonstrate our loss function's usefulness on Fashion MNIST, CIFAR-10, CIFAR-100 datasets, where we heuristically create training data with noisy labels and achieve a nice performance boost compared to the standard binary cross-entropy. These results are also confirmed in the second experiment, where we use a trained model on Google Images to classify the ImageWoof dataset, and the third experiment, where we deal with the WebVision and ANIMAL-10N datasets. We also show that the proposed technique yields significantly better performance than the gradient clipping. Code: gitlab.com/irafmai/clipping_cross_entropy

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10200 - Computer and information sciences

Project

<a href="/en/project/EF17_049%2F0008414" target="_blank" >EF17_049/0008414: Centre for the development of Artificial Intelligence Methods for the Automotive Industry of the region</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2022

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

NEURAL COMPUT APPL

ISSN

0941-0643

e-ISSN

1433-3058

Volume of the periodical

—

Issue of the periodical within the volume

14

Country of publishing house

GB - UNITED KINGDOM

Number of pages

13

Pages from-to

12029-12041

UT code for WoS article

000780838900001

EID of the result in the Scopus database

2-s2.0-85128058703