Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F21%3A00357178" target="_blank" >RIV/68407700:21230/21:00357178 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1007/978-3-030-92659-5_7" target="_blank" >https://doi.org/10.1007/978-3-030-92659-5_7</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-030-92659-5_7" target="_blank" >10.1007/978-3-030-92659-5_7</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks

Popis výsledku v původním jazyce

Training neural networks with binary weights and activa-tions is a challenging problem due to the lack of gradients and difficulty ofoptimization over discrete weights. Many successful experimental resultshave been achieved with empirical straight-through (ST) approaches,proposing a variety of ad-hoc rules for propagating gradients throughnon-differentiable activations and updating discrete weights. At the sametime, ST methods can be truly derived as estimators in the stochasticbinary network (SBN) model with Bernoulli weights. We advance thesederivations to a more complete and systematic study. We analyze proper-ties, estimation accuracy, obtain different forms of correct ST estimatorsfor activations and weights, explain existing empirical approaches andtheir shortcomings, explain how latent weights arise from the mirrordescent method when optimizing over probabilities. This allows to rein-troduce ST methods, long known empirically, as sound approximations,apply them with clarity and develop further improvements.

Název v anglickém jazyce

Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks

Popis výsledku anglicky

Training neural networks with binary weights and activa-tions is a challenging problem due to the lack of gradients and difficulty ofoptimization over discrete weights. Many successful experimental resultshave been achieved with empirical straight-through (ST) approaches,proposing a variety of ad-hoc rules for propagating gradients throughnon-differentiable activations and updating discrete weights. At the sametime, ST methods can be truly derived as estimators in the stochasticbinary network (SBN) model with Bernoulli weights. We advance thesederivations to a more complete and systematic study. We analyze proper-ties, estimation accuracy, obtain different forms of correct ST estimatorsfor activations and weights, explain existing empirical approaches andtheir shortcomings, explain how latent weights arise from the mirrordescent method when optimizing over probabilities. This allows to rein-troduce ST methods, long known empirically, as sound approximations,apply them with clarity and develop further improvements.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Výzkumné centrum informatiky</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 43rd DAGM German Conference (DAGM GCPR 2021)

ISBN

978-3-030-92658-8

ISSN

0302-9743

e-ISSN

1611-3349

Počet stran výsledku

16

Strana od-do

111-126

Název nakladatele

Springer

Místo vydání

Cham

Místo konání akce

Bonn

Datum konání akce

28. 9. 2021

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—