Cold Analysis of Rao-Blackwellized Straight-Through Gumbel-Softmax Gradient Estimator

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F23%3A00371050" target="_blank" >RIV/68407700:21230/23:00371050 - isvavai.cz</a>

Result on the web

<a href="https://proceedings.mlr.press/v202/shekhovtsov23a.html" target="_blank" >https://proceedings.mlr.press/v202/shekhovtsov23a.html</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Cold Analysis of Rao-Blackwellized Straight-Through Gumbel-Softmax Gradient Estimator

Original language description

Many problems in machine learning require an estimate of the gradient of an expectation in discrete random variables with respect to the sampling distribution. This work is motivated by the development of the Gumbel-Softmax family of estimators, which use a temperature-controlled relaxation of discrete variables. The state-of-the art in this family, the Gumbel-Rao estimator uses an extra internal sampling to reduce the variance, which may be costly. We analyze this estimator and show that it possesses a zero temperature limit with a surprisingly simple closed form. The limit estimator, called ZGR, has favorable bias and variance properties, it is easy to implement and computationally inexpensive. It decomposes as the average of the straight through (ST) estimator and DARN estimator — two basic but not very well performing on their own estimators. We demonstrate that the simple ST–ZGR family of estimators practically dominates in the bias-variance tradeoffs the whole GR family while also outperforming SOTA unbiased estimators.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Research Center for Informatics</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

International Conference on Machine Learning (ICML)

ISBN

—

ISSN

2640-3498

e-ISSN

2640-3498

Number of pages

25

Pages from-to

30931-30955

Publisher name

Proceedings of Machine Learning Research

Place of publication

—

Event location

Honolulu, USA

Event date

Jul 23, 2023

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—