HyperQueue: Efficient and ergonomic task graphs on HPC clusters

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989100%3A27740%2F24%3A10255113" target="_blank" >RIV/61989100:27740/24:10255113 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.softxjournal.com/article/S2352-7110(24)00185-7/fulltext" target="_blank" >https://www.softxjournal.com/article/S2352-7110(24)00185-7/fulltext</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.softx.2024.101814" target="_blank" >10.1016/j.softx.2024.101814</a>

Jazyk výsledku

angličtina

Název v původním jazyce

HyperQueue: Efficient and ergonomic task graphs on HPC clusters

Popis výsledku v původním jazyce

Task graphs are a popular method for defining complex scientific simulations and experiments that run on distributed and HPC (High-performance computing) clusters, because they allow their authors to focus on the problem domain, instead of low-level communication between nodes, and also enable quick prototyping. However, executing task graphs on HPC clusters can be problematic in the presence of allocation managers like PBS or Slurm, which are not designed for executing a large number of potentially short-lived tasks with dependencies. To make task graph execution on HPC clusters more efficient and ergonomic, we have created HYPERQUEUE, an open-source task graph execution runtime tailored for HPC use-cases. It enables the execution of large task graphs on top of an allocation manager by aggregating tasks into a smaller amount of PBS/Slurm allocations and dynamically load balances tasks amongst all available nodes. It can also automatically submit allocations on behalf of the user, it supports arbitrary task resource requirements and heterogeneous HPC clusters, it is trivial to deploy and does not require elevated privileges.

Název v anglickém jazyce

HyperQueue: Efficient and ergonomic task graphs on HPC clusters

Popis výsledku anglicky

Task graphs are a popular method for defining complex scientific simulations and experiments that run on distributed and HPC (High-performance computing) clusters, because they allow their authors to focus on the problem domain, instead of low-level communication between nodes, and also enable quick prototyping. However, executing task graphs on HPC clusters can be problematic in the presence of allocation managers like PBS or Slurm, which are not designed for executing a large number of potentially short-lived tasks with dependencies. To make task graph execution on HPC clusters more efficient and ergonomic, we have created HYPERQUEUE, an open-source task graph execution runtime tailored for HPC use-cases. It enables the execution of large task graphs on top of an allocation manager by aggregating tasks into a smaller amount of PBS/Slurm allocations and dynamically load balances tasks amongst all available nodes. It can also automatically submit allocations on behalf of the user, it supports arbitrary task resource requirements and heterogeneous HPC clusters, it is trivial to deploy and does not require elevated privileges.

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

V - Vyzkumna aktivita podporovana z jinych verejnych zdroju

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

SoftwareX

ISSN

2352-7110

e-ISSN

—

Svazek periodika

27

Číslo periodika v rámci svazku

September

Stát vydavatele periodika

NL - Nizozemsko

Počet stran výsledku

6

Strana od-do

1-6

Kód UT WoS článku

001267389800001

EID výsledku v databázi Scopus

2-s2.0-85198121697