Cross-Entropy Loss of Approximated Deep Neural Networks

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F25%3A00638227" target="_blank" >RIV/67985807:_____/25:00638227 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1007/978-981-95-4367-0_31" target="_blank" >https://doi.org/10.1007/978-981-95-4367-0_31</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-981-95-4367-0_31" target="_blank" >10.1007/978-981-95-4367-0_31</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Cross-Entropy Loss of Approximated Deep Neural Networks

Popis výsledku v původním jazyce

Deep neural networks (DNNs), which underpin modern AI technologies, demand substantial computational resources, posing challenges for deployment on energy-constrained devices (e.g., battery-powered smartphones). A viable solution is to reduce the complexity of trained DNNs via approximate computing techniques, such as low-bit quantization or pruning, which significantly lower energy consumption with minimal impact on inference accuracy. In this paper, we adapt our AppMax method—originally developed for estimating regression error of approximated neural networks (NNs)—to upper-bound the cross-entropy loss between the output categorical probability distributions of a trained classification DNN with softmax (e.g., a convolutional NN) and its lowenergy approximation. Using the concept of shortcut weights and optimal linear interpolation of the exponential function, AppMax bounds this loss via linear programming over convex polytopes around test/training data points, constrained to regions where the same category is originally inferred with high probability. Preliminary MNIST experiments show that AppMax identifies inputs with maximum cross-entropy loss, some of which are misclassified by the approximated NN (with reduced weight bitwidth), even though its overall accuracy on the test data is preserved. This error bound can be used to evaluate different approximation strategies and identify those that best balance accuracy and energy efficiency.

Název v anglickém jazyce

Cross-Entropy Loss of Approximated Deep Neural Networks

Popis výsledku anglicky

Deep neural networks (DNNs), which underpin modern AI technologies, demand substantial computational resources, posing challenges for deployment on energy-constrained devices (e.g., battery-powered smartphones). A viable solution is to reduce the complexity of trained DNNs via approximate computing techniques, such as low-bit quantization or pruning, which significantly lower energy consumption with minimal impact on inference accuracy. In this paper, we adapt our AppMax method—originally developed for estimating regression error of approximated neural networks (NNs)—to upper-bound the cross-entropy loss between the output categorical probability distributions of a trained classification DNN with softmax (e.g., a convolutional NN) and its lowenergy approximation. Using the concept of shortcut weights and optimal linear interpolation of the exponential function, AppMax bounds this loss via linear programming over convex polytopes around test/training data points, constrained to regions where the same category is originally inferred with high probability. Preliminary MNIST experiments show that AppMax identifies inputs with maximum cross-entropy loss, some of which are misclassified by the approximated NN (with reduced weight bitwidth), even though its overall accuracy on the test data is preserved. This error bound can be used to evaluate different approximation strategies and identify those that best balance accuracy and energy efficiency.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GA25-15490S" target="_blank" >GA25-15490S: LEDNeCo: Nízkoenergetické hluboké neurovýpočty</a><br>

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Neural Information Processing. ICONIP 2025 Proceedings, Part I

ISBN

978-981-95-4366-3

ISSN

0302-9743

e-ISSN

—

Počet stran výsledku

16

Strana od-do

460-475

Název nakladatele

Springer

Místo vydání

Cham

Místo konání akce

Okinawa

Datum konání akce

20. 11. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—