Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F21%3A00357180" target="_blank" >RIV/68407700:21230/21:00357180 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1007/978-3-030-92659-5_8" target="_blank" >https://doi.org/10.1007/978-3-030-92659-5_8</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/978-3-030-92659-5_8" target="_blank" >10.1007/978-3-030-92659-5_8</a>
Alternative languages
Result language
angličtina
Original language name
Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators
Original language description
Discrete and especially binary random variables occur in many machine learning models, notably in variational autoencoders with binary latent states and in stochastic binary networks. When learning such models, a key tool is an estimator of the gradient of the expected loss with respect to the probabilities of binary variables. The straight-through (ST) estimator gained popularity due to its simplicity and efficiency, in particular in deep networks where unbiased estimators are impractical. Several techniques were proposed to improve over ST while keeping the same low computational complexity: Gumbel-Softmax, ST-Gumbel-Softmax, BayesBiNN, FouST. We conduct a theoretical analysis of bias and variance of these methods in order to understand tradeoffs and verify the originally claimed properties. The presented theoretical results allow for better understanding of these methods and in some cases reveal serious issues.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Research Center for Informatics</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 43rd DAGM German Conference (DAGM GCPR 2021)
ISBN
978-3-030-92658-8
ISSN
0302-9743
e-ISSN
1611-3349
Number of pages
15
Pages from-to
127-141
Publisher name
Springer
Place of publication
Cham
Event location
Bonn
Event date
Sep 28, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—