Binary cross-entropy with dynamical clipping

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61988987%3A17610%2F22%3AA2302717" target="_blank" >RIV/61988987:17610/22:A2302717 - isvavai.cz</a>

Výsledek na webu

<a href="https://link.springer.com/article/10.1007/s00521-022-07091-x" target="_blank" >https://link.springer.com/article/10.1007/s00521-022-07091-x</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s00521-022-07091-x" target="_blank" >10.1007/s00521-022-07091-x</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Binary cross-entropy with dynamical clipping

Popis výsledku v původním jazyce

We investigate the adverse effect of noisy labels in a training dataset on a neural network's precision in an image classification task. The importance of this research lies in the fact that most datasets include noisy labels. To reduce the impact of noisy labels, we propose to extend the binary cross-entropy by dynamical clipping, which clips all samples' loss values in a mini-batch by a clipping constant. Such a constant is dynamically determined for every single mini-batch using its statistics. The advantage is the dynamic adaptation to any number of noisy labels in a training dataset. Thanks to that, the proposed binary cross-entropy with dynamical clipping can be used in any model utilizing cross-entropy or focal loss, including pre-trained models. We prove that the proposed loss function is an alpha-calibrated classification loss, implying consistency and robustness to noise misclassification in more general asymmetric problems. We demonstrate our loss function's usefulness on Fashion MNIST, CIFAR-10, CIFAR-100 datasets, where we heuristically create training data with noisy labels and achieve a nice performance boost compared to the standard binary cross-entropy. These results are also confirmed in the second experiment, where we use a trained model on Google Images to classify the ImageWoof dataset, and the third experiment, where we deal with the WebVision and ANIMAL-10N datasets. We also show that the proposed technique yields significantly better performance than the gradient clipping. Code: gitlab.com/irafmai/clipping_cross_entropy

Název v anglickém jazyce

Binary cross-entropy with dynamical clipping

Popis výsledku anglicky

We investigate the adverse effect of noisy labels in a training dataset on a neural network's precision in an image classification task. The importance of this research lies in the fact that most datasets include noisy labels. To reduce the impact of noisy labels, we propose to extend the binary cross-entropy by dynamical clipping, which clips all samples' loss values in a mini-batch by a clipping constant. Such a constant is dynamically determined for every single mini-batch using its statistics. The advantage is the dynamic adaptation to any number of noisy labels in a training dataset. Thanks to that, the proposed binary cross-entropy with dynamical clipping can be used in any model utilizing cross-entropy or focal loss, including pre-trained models. We prove that the proposed loss function is an alpha-calibrated classification loss, implying consistency and robustness to noise misclassification in more general asymmetric problems. We demonstrate our loss function's usefulness on Fashion MNIST, CIFAR-10, CIFAR-100 datasets, where we heuristically create training data with noisy labels and achieve a nice performance boost compared to the standard binary cross-entropy. These results are also confirmed in the second experiment, where we use a trained model on Google Images to classify the ImageWoof dataset, and the third experiment, where we deal with the WebVision and ANIMAL-10N datasets. We also show that the proposed technique yields significantly better performance than the gradient clipping. Code: gitlab.com/irafmai/clipping_cross_entropy

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10200 - Computer and information sciences

Projekt

<a href="/cs/project/EF17_049%2F0008414" target="_blank" >EF17_049/0008414: Centrum pro výzkum a vývoj metod umělé intelligence v automobilovém průmyslu regionu</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2022

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

NEURAL COMPUT APPL

ISSN

0941-0643

e-ISSN

1433-3058

Svazek periodika

—

Číslo periodika v rámci svazku

14

Stát vydavatele periodika

GB - Spojené království Velké Británie a Severního Irska

Počet stran výsledku

13

Strana od-do

12029-12041

Kód UT WoS článku

000780838900001

EID výsledku v databázi Scopus

2-s2.0-85128058703