Mean-field Analysis for Heavy Ball Methods: Dropout-stability, Connectivity, and Global Convergence

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F23%3A00375312" target="_blank" >RIV/68407700:21230/23:00375312 - isvavai.cz</a>

Výsledek na webu

<a href="https://openreview.net/pdf?id=gZna3IiGfl" target="_blank" >https://openreview.net/pdf?id=gZna3IiGfl</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Mean-field Analysis for Heavy Ball Methods: Dropout-stability, Connectivity, and Global Convergence

Popis výsledku v původním jazyce

The stochastic heavy ball method (SHB), also known as stochastic gradient descent (SGD) with Polyak's momentum, is widely used in training neural networks. However, despite the remarkable success of such algorithm in practice, its theoretical characterization remains limited. In this paper, we focus on neural networks with two and three layers and provide a rigorous understanding of the properties of the solutions found by SHB: emph{(i)} stability after dropping out part of the neurons, emph{(ii)} connectivity along a low-loss path, and emph{(iii)} convergence to the global optimum. To achieve this goal, we take a mean-field view and relate the SHB dynamics to a certain partial differential equation in the limit of large network widths. This mean-field perspective has inspired a recent line of work focusing on SGD while, in contrast, our paper considers an algorithm with momentum. More specifically, after proving existence and uniqueness of the limit differential equations, we show convergence to the global optimum and give a quantitative bound between the mean-field limit and the SHB dynamics of a finite-width network. Armed with this last bound, we are able to establish the dropout-stability and connectivity of SHB solutions.

Název v anglickém jazyce

Mean-field Analysis for Heavy Ball Methods: Dropout-stability, Connectivity, and Global Convergence

Popis výsledku anglicky

The stochastic heavy ball method (SHB), also known as stochastic gradient descent (SGD) with Polyak's momentum, is widely used in training neural networks. However, despite the remarkable success of such algorithm in practice, its theoretical characterization remains limited. In this paper, we focus on neural networks with two and three layers and provide a rigorous understanding of the properties of the solutions found by SHB: emph{(i)} stability after dropping out part of the neurons, emph{(ii)} connectivity along a low-loss path, and emph{(iii)} convergence to the global optimum. To achieve this goal, we take a mean-field view and relate the SHB dynamics to a certain partial differential equation in the limit of large network widths. This mean-field perspective has inspired a recent line of work focusing on SGD while, in contrast, our paper considers an algorithm with momentum. More specifically, after proving existence and uniqueness of the limit differential equations, we show convergence to the global optimum and give a quantitative bound between the mean-field limit and the SHB dynamics of a finite-width network. Armed with this last bound, we are able to establish the dropout-stability and connectivity of SHB solutions.

Druh

J_ost - Ostatní články v recenzovaných periodicích

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Výzkumné centrum informatiky</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Transactions on Machine Learning Research

ISSN

2835-8856

e-ISSN

2835-8856

Svazek periodika

—

Číslo periodika v rámci svazku

February

Stát vydavatele periodika

DE - Spolková republika Německo

Počet stran výsledku

49

Strana od-do

1-49

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-105000206429