Cross-Entropy Loss of Approximated Deep Neural Networks

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F25%3A00638227" target="_blank" >RIV/67985807:_____/25:00638227 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1007/978-981-95-4367-0_31" target="_blank" >https://doi.org/10.1007/978-981-95-4367-0_31</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-981-95-4367-0_31" target="_blank" >10.1007/978-981-95-4367-0_31</a>

Result language

angličtina

Original language name

Cross-Entropy Loss of Approximated Deep Neural Networks

Original language description

Deep neural networks (DNNs), which underpin modern AI technologies, demand substantial computational resources, posing challenges for deployment on energy-constrained devices (e.g., battery-powered smartphones). A viable solution is to reduce the complexity of trained DNNs via approximate computing techniques, such as low-bit quantization or pruning, which significantly lower energy consumption with minimal impact on inference accuracy. In this paper, we adapt our AppMax method—originally developed for estimating regression error of approximated neural networks (NNs)—to upper-bound the cross-entropy loss between the output categorical probability distributions of a trained classification DNN with softmax (e.g., a convolutional NN) and its lowenergy approximation. Using the concept of shortcut weights and optimal linear interpolation of the exponential function, AppMax bounds this loss via linear programming over convex polytopes around test/training data points, constrained to regions where the same category is originally inferred with high probability. Preliminary MNIST experiments show that AppMax identifies inputs with maximum cross-entropy loss, some of which are misclassified by the approximated NN (with reduced weight bitwidth), even though its overall accuracy on the test data is preserved. This error bound can be used to evaluate different approximation strategies and identify those that best balance accuracy and energy efficiency.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/GA25-15490S" target="_blank" >GA25-15490S: LEDNeCo: Low Energy Deep Neurocomputing</a><br>

Continuities

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Neural Information Processing. ICONIP 2025 Proceedings, Part I

ISBN

978-981-95-4366-3

ISSN

0302-9743

e-ISSN

—

Number of pages

16

Pages from-to

460-475

Publisher name

Springer

Place of publication

Cham

Event location

Okinawa

Event date

Nov 20, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—