Initialization and Transfer Learning of Stochastic Binary Networks from Real-Valued Ones
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F21%3A00357000" target="_blank" >RIV/68407700:21230/21:00357000 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1109/CVPRW53098.2021.00524" target="_blank" >https://doi.org/10.1109/CVPRW53098.2021.00524</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/CVPRW53098.2021.00524" target="_blank" >10.1109/CVPRW53098.2021.00524</a>
Alternative languages
Result language
angličtina
Original language name
Initialization and Transfer Learning of Stochastic Binary Networks from Real-Valued Ones
Original language description
We consider the training of binary neural networks (BNNs) using the stochastic relaxation approach, which leads to stochastic binary networks (SBNs). We identify that a severe obstacle to training deep SBNs without skip connections is already the initialization phase. While smaller models can be trained from a random (possibly data-driven) initialization, for deeper models and large datasets, it becomes increasingly difficult to obtain non-vanishing and low variance gradients when initializing randomly.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Research Center for Informatics</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGITION WORKSHOPS (CVPRW 2021)
ISBN
978-1-6654-4899-4
ISSN
2160-7508
e-ISSN
2160-7516
Number of pages
9
Pages from-to
4655-4663
Publisher name
IEEE Computer Society Press
Place of publication
New York
Event location
Nashville
Event date
Jun 20, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
000705890204090