A Comparison of Regularization Techniques for Shallow Neural Networks Trained on Small Datasets
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F21%3A00546161" target="_blank" >RIV/67985807:_____/21:00546161 - isvavai.cz</a>
Result on the web
<a href="https://ics.upjs.sk/~antoni/ceur-ws.org/Vol-0000/paper38.pdf" target="_blank" >https://ics.upjs.sk/~antoni/ceur-ws.org/Vol-0000/paper38.pdf</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
A Comparison of Regularization Techniques for Shallow Neural Networks Trained on Small Datasets
Original language description
Neural networks are frequently used as regression models. Their training is usually difficult when the model is subject to a small training dataset with numerous outliers. This paper investigates the effects of various regularisation techniques that can help with this kind of problem. We analysed the effects of the model size, loss selection, L2 weight regularisation, L2 activity regularisation, Dropout, and Alpha Dropout. We collected 30 different datasets, each of which has been split by ten-fold cross-validation. As an evaluation metric, we used cumulative distribution functions (CDFs) of L1 and L2 losses to aggregate results from different datasets without a considerable amount of distortion. Distributions of the metrics are shown, and thorough statistical tests were conducted. Surprisingly, the results show that Dropout models are not suited for our objective. The most effective approach is the choice of model size and L2 types of regularisations.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 21st Conference Information Technologies – Applications and Theory (ITAT 2021)
ISBN
—
ISSN
1613-0073
e-ISSN
—
Number of pages
10
Pages from-to
94-103
Publisher name
Technical University & CreateSpace Independent Publishing
Place of publication
Aachen
Event location
Heľpa
Event date
Sep 24, 2021
Type of event by nationality
EUR - Evropská akce
UT code for WoS article
—