Runtime vs Scheduler: Analyzing Dask's Overheads
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989100%3A27740%2F20%3A10245524" target="_blank" >RIV/61989100:27740/20:10245524 - isvavai.cz</a>
Result on the web
<a href="https://ieeexplore.ieee.org/abstract/document/9308114" target="_blank" >https://ieeexplore.ieee.org/abstract/document/9308114</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/WORKS51914.2020.00006" target="_blank" >10.1109/WORKS51914.2020.00006</a>
Alternative languages
Result language
angličtina
Original language name
Runtime vs Scheduler: Analyzing Dask's Overheads
Original language description
Dask is a distributed task framework which is commonly used by data scientists to parallelize Python code on computing clusters with little programming effort. It uses a sophisticated work-stealing scheduler which has been hand-tuned to execute task graphs as efficiently as possible. But is scheduler optimization a worthwhile effort for Dask? Our paper shows on many real world task graphs that even a completely random scheduler is surprisingly competitive with its built-in scheduler and that the main bottleneck of Dask lies in its runtime overhead. We develop a drop-in replacement for the Dask central server written in Rust which is backwards compatible with existing Dask programs. Thanks to its efficient runtime, our server implementation is able to scale up to larger clusters than Dask and consistently outperforms it on a variety of task graphs, despite the fact that it uses a simpler scheduling algorithm.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach
Others
Publication year
2020
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of 15th Workshop on Workflows in Support of Large-Scale Science, WORKS 2020 : November 2020
ISBN
978-0-7381-1040-0
ISSN
—
e-ISSN
—
Number of pages
8
Pages from-to
1-8
Publisher name
IEEE - Institute of Electrical and Electronics Engineers
Place of publication
Piscataway
Event location
Atlanta
Event date
Nov 11, 2020
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—