Robust and resource efficient identification of shallow neural networks by fewest samples

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21340%2F21%3A00353191" target="_blank" >RIV/68407700:21340/21:00353191 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1093/imaiai/iaaa036" target="_blank" >https://doi.org/10.1093/imaiai/iaaa036</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1093/imaiai/iaaa036" target="_blank" >10.1093/imaiai/iaaa036</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Robust and resource efficient identification of shallow neural networks by fewest samples

Popis výsledku v původním jazyce

We address the structure identification and the uniform approximation of sums of ridge functions f (x) = Sigma(m)(i=1) g(i)(< a(i), x >) on R-d, representing a general form of a shallow feed-forward neural network, from a small number of query samples. Higher order differentiation, as used in our constructive approximations, of sums of ridge functions or of their compositions, as in deeper neural network, yields a natural connection between neural network weight identification and tensor product decomposition identification. In the case of the shallowest feed-forward neural network, second-order differentiation and tensors of order two (i.e., matrices) suffice as we prove in this paper. We use two sampling schemes to perform approximate differentiation-active sampling, where the sampling points are universal, actively and randomly designed, and passive sampling, where sampling points were preselected at random from a distribution with known density. Based on multiple gathered approximated first- and second-order differentials, our general approximation strategy is developed as a sequence of algorithms to perform individual sub-tasks. We first perform an active subspace search by approximating the span of the weight vectors a(1),( ...), a(m). Then we use a straightforward substitution, which reduces the dimensionality of the problem from d to m. The core of the construction is then the stable and efficient approximation of weights expressed in terms of rank-1 matrices ai circle times ai, realized by formulating their individual identification as a suitable nonlinear program. We prove the successful identification by this program of weight vectors being close to orthonormal and we also show how we can constructively reduce to this case by a whitening procedure, without loss of any generality. We finally discuss the implementation and the performance of the proposed algorithmic pipeline with extensive numerical experiments, which illustrate and confirm the theoretical

Název v anglickém jazyce

Robust and resource efficient identification of shallow neural networks by fewest samples

Popis výsledku anglicky

We address the structure identification and the uniform approximation of sums of ridge functions f (x) = Sigma(m)(i=1) g(i)(< a(i), x >) on R-d, representing a general form of a shallow feed-forward neural network, from a small number of query samples. Higher order differentiation, as used in our constructive approximations, of sums of ridge functions or of their compositions, as in deeper neural network, yields a natural connection between neural network weight identification and tensor product decomposition identification. In the case of the shallowest feed-forward neural network, second-order differentiation and tensors of order two (i.e., matrices) suffice as we prove in this paper. We use two sampling schemes to perform approximate differentiation-active sampling, where the sampling points are universal, actively and randomly designed, and passive sampling, where sampling points were preselected at random from a distribution with known density. Based on multiple gathered approximated first- and second-order differentials, our general approximation strategy is developed as a sequence of algorithms to perform individual sub-tasks. We first perform an active subspace search by approximating the span of the weight vectors a(1),( ...), a(m). Then we use a straightforward substitution, which reduces the dimensionality of the problem from d to m. The core of the construction is then the stable and efficient approximation of weights expressed in terms of rank-1 matrices ai circle times ai, realized by formulating their individual identification as a suitable nonlinear program. We prove the successful identification by this program of weight vectors being close to orthonormal and we also show how we can constructively reduce to this case by a whitening procedure, without loss of any generality. We finally discuss the implementation and the performance of the proposed algorithmic pipeline with extensive numerical experiments, which illustrate and confirm the theoretical

Druh

J_ost - Ostatní články v recenzovaných periodicích

CEP obor

—

OECD FORD obor

10101 - Pure mathematics

Projekt

<a href="/cs/project/8X20043" target="_blank" >8X20043: Časově frekvenční reprezentace prostoru funkcí</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Information and Inference: a Journal of the IMA

ISSN

2049-8772

e-ISSN

2049-8772

Svazek periodika

10

Číslo periodika v rámci svazku

2

Stát vydavatele periodika

GB - Spojené království Velké Británie a Severního Irska

Počet stran výsledku

71

Strana od-do

625-695

Kód UT WoS článku

000670949400008

EID výsledku v databázi Scopus

—