HyperQueue: Efficient and ergonomic task graphs on HPC clusters
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989100%3A27740%2F24%3A10255113" target="_blank" >RIV/61989100:27740/24:10255113 - isvavai.cz</a>
Result on the web
<a href="https://www.softxjournal.com/article/S2352-7110(24)00185-7/fulltext" target="_blank" >https://www.softxjournal.com/article/S2352-7110(24)00185-7/fulltext</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1016/j.softx.2024.101814" target="_blank" >10.1016/j.softx.2024.101814</a>
Alternative languages
Result language
angličtina
Original language name
HyperQueue: Efficient and ergonomic task graphs on HPC clusters
Original language description
Task graphs are a popular method for defining complex scientific simulations and experiments that run on distributed and HPC (High-performance computing) clusters, because they allow their authors to focus on the problem domain, instead of low-level communication between nodes, and also enable quick prototyping. However, executing task graphs on HPC clusters can be problematic in the presence of allocation managers like PBS or Slurm, which are not designed for executing a large number of potentially short-lived tasks with dependencies. To make task graph execution on HPC clusters more efficient and ergonomic, we have created HYPERQUEUE, an open-source task graph execution runtime tailored for HPC use-cases. It enables the execution of large task graphs on top of an allocation manager by aggregating tasks into a smaller amount of PBS/Slurm allocations and dynamically load balances tasks amongst all available nodes. It can also automatically submit allocations on behalf of the user, it supports arbitrary task resource requirements and heterogeneous HPC clusters, it is trivial to deploy and does not require elevated privileges.
Czech name
—
Czech description
—
Classification
Type
J<sub>imp</sub> - Article in a specialist periodical, which is included in the Web of Science database
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
V - Vyzkumna aktivita podporovana z jinych verejnych zdroju
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Name of the periodical
SoftwareX
ISSN
2352-7110
e-ISSN
—
Volume of the periodical
27
Issue of the periodical within the volume
September
Country of publishing house
NL - THE KINGDOM OF THE NETHERLANDS
Number of pages
6
Pages from-to
1-6
UT code for WoS article
001267389800001
EID of the result in the Scopus database
2-s2.0-85198121697