Towards Efficient Scheduling of Transformer Neural Network Computation for Edge AI Deployment

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26230%2F26%3A0197537" target="_blank" >RIV/00216305:26230/26:0197537 - isvavai.cz</a>

Výsledek na webu

<a href="http://dx.doi.org/10.1145/3712255.3734345" target="_blank" >http://dx.doi.org/10.1145/3712255.3734345</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3712255.3734345" target="_blank" >10.1145/3712255.3734345</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Towards Efficient Scheduling of Transformer Neural Network Computation for Edge AI Deployment

Popis výsledku v původním jazyce

Transformer neural networks have gained popularity in recent years, demonstrating remarkable performance across many application domains. However, inference on resource-constrained embedded hardware remains challenging due to Transformers' substantial computational demands. We aim to address this problem by focusing on exploiting the inherent parallelism opportunities presented by the multi-head self attention operations of Transformers, to achieve a speedup in processing on embedded hardware. In this paper, we present an evolutionary-based scheduling approach for distribution and allocation of Transformer operations across systolic array-based hardware accelerators used for execution. Our methodology takes as input specifications of the Transformer workload and the target systolic array architecture and explores the large mapping space to identify an efficient plan of operation-to-array assignments. The plans are evaluated against a hardware-aware cost model, capturing the cost of computational cycles for a given operation and systolic array, with the objective to minimize the total sum across all operations. Through extensive experimental evaluations across diverse systolic array dimensions, we demonstrate that our evolutionary-based scheduler surpasses conventional heuristics and is able to find plans offering up to 33.8% average reduction in overall cycle count.

Název v anglickém jazyce

Towards Efficient Scheduling of Transformer Neural Network Computation for Edge AI Deployment

Popis výsledku anglicky

Transformer neural networks have gained popularity in recent years, demonstrating remarkable performance across many application domains. However, inference on resource-constrained embedded hardware remains challenging due to Transformers' substantial computational demands. We aim to address this problem by focusing on exploiting the inherent parallelism opportunities presented by the multi-head self attention operations of Transformers, to achieve a speedup in processing on embedded hardware. In this paper, we present an evolutionary-based scheduling approach for distribution and allocation of Transformer operations across systolic array-based hardware accelerators used for execution. Our methodology takes as input specifications of the Transformer workload and the target systolic array architecture and explores the large mapping space to identify an efficient plan of operation-to-array assignments. The plans are evaluated against a hardware-aware cost model, capturing the cost of computational cycles for a given operation and systolic array, with the objective to minimize the total sum across all operations. Through extensive experimental evaluations across diverse systolic array dimensions, we demonstrate that our evolutionary-based scheduler surpasses conventional heuristics and is able to find plans offering up to 33.8% average reduction in overall cycle count.

Druh

O - Ostatní výsledky

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GA25-15490S" target="_blank" >GA25-15490S: LEDNeCo: Nízkoenergetické hluboké neurovýpočty</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů