Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F21%3A00357180" target="_blank" >RIV/68407700:21230/21:00357180 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1007/978-3-030-92659-5_8" target="_blank" >https://doi.org/10.1007/978-3-030-92659-5_8</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-030-92659-5_8" target="_blank" >10.1007/978-3-030-92659-5_8</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators

Popis výsledku v původním jazyce

Discrete and especially binary random variables occur in many machine learning models, notably in variational autoencoders with binary latent states and in stochastic binary networks. When learning such models, a key tool is an estimator of the gradient of the expected loss with respect to the probabilities of binary variables. The straight-through (ST) estimator gained popularity due to its simplicity and efficiency, in particular in deep networks where unbiased estimators are impractical. Several techniques were proposed to improve over ST while keeping the same low computational complexity: Gumbel-Softmax, ST-Gumbel-Softmax, BayesBiNN, FouST. We conduct a theoretical analysis of bias and variance of these methods in order to understand tradeoffs and verify the originally claimed properties. The presented theoretical results allow for better understanding of these methods and in some cases reveal serious issues.

Název v anglickém jazyce

Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators

Popis výsledku anglicky

Discrete and especially binary random variables occur in many machine learning models, notably in variational autoencoders with binary latent states and in stochastic binary networks. When learning such models, a key tool is an estimator of the gradient of the expected loss with respect to the probabilities of binary variables. The straight-through (ST) estimator gained popularity due to its simplicity and efficiency, in particular in deep networks where unbiased estimators are impractical. Several techniques were proposed to improve over ST while keeping the same low computational complexity: Gumbel-Softmax, ST-Gumbel-Softmax, BayesBiNN, FouST. We conduct a theoretical analysis of bias and variance of these methods in order to understand tradeoffs and verify the originally claimed properties. The presented theoretical results allow for better understanding of these methods and in some cases reveal serious issues.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Výzkumné centrum informatiky</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 43rd DAGM German Conference (DAGM GCPR 2021)

ISBN

978-3-030-92658-8

ISSN

0302-9743

e-ISSN

1611-3349

Počet stran výsledku

15

Strana od-do

127-141

Název nakladatele

Springer

Místo vydání

Cham

Místo konání akce

Bonn

Datum konání akce

28. 9. 2021

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—