Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F22%3A00358262" target="_blank" >RIV/68407700:21230/22:00358262 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.48550/arXiv.2111.02278" target="_blank" >https://doi.org/10.48550/arXiv.2111.02278</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.48550/arXiv.2111.02278" target="_blank" >10.48550/arXiv.2111.02278</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks

Popis výsledku v původním jazyce

Understanding the properties of neural networks trained via stochastic gradient descent (SGD) is at the heart of the theory of deep learning. In this work, we take a mean- field view, and consider a two-layer ReLU network trained via noisy-SGD for a univariate regularized regression problem. Our main result is that SGD with vanishingly small noise injected in the gradients is biased towards a simple solution: at convergence, the ReLU network implements a piecewise linear map of the inputs, and the number of knot"points { i.e., points where the tangent of the ReLU network estimator changes { between two consecutive training inputs is at most three. In particular, as the number of neurons of the network grows, the SGD dynamics is captured by the solution of a gradient ow and, at convergence, the distribution of the weights approaches the unique minimizer of a related free energy, which has a Gibbs form. Our key technical contribution consists in the analysis of the estimator resulting from this minimizer: we show that its second derivative vanishes everywhere, except at some specific locations which represent the knot"points. We also provide empirical evidence that knots at locations distinct from the data points might occur, as predicted by our theory.

Název v anglickém jazyce

Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks

Popis výsledku anglicky

Understanding the properties of neural networks trained via stochastic gradient descent (SGD) is at the heart of the theory of deep learning. In this work, we take a mean- field view, and consider a two-layer ReLU network trained via noisy-SGD for a univariate regularized regression problem. Our main result is that SGD with vanishingly small noise injected in the gradients is biased towards a simple solution: at convergence, the ReLU network implements a piecewise linear map of the inputs, and the number of knot"points { i.e., points where the tangent of the ReLU network estimator changes { between two consecutive training inputs is at most three. In particular, as the number of neurons of the network grows, the SGD dynamics is captured by the solution of a gradient ow and, at convergence, the distribution of the weights approaches the unique minimizer of a related free energy, which has a Gibbs form. Our key technical contribution consists in the analysis of the estimator resulting from this minimizer: we show that its second derivative vanishes everywhere, except at some specific locations which represent the knot"points. We also provide empirical evidence that knots at locations distinct from the data points might occur, as predicted by our theory.

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2022

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Journal of Machine Learning Research

ISSN

1532-4435

e-ISSN

—

Svazek periodika

23

Číslo periodika v rámci svazku

130

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

55

Strana od-do

1-55

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-85130359653