Runtime vs Scheduler: Analyzing Dask's Overheads

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989100%3A27740%2F20%3A10245524" target="_blank" >RIV/61989100:27740/20:10245524 - isvavai.cz</a>

Výsledek na webu

<a href="https://ieeexplore.ieee.org/abstract/document/9308114" target="_blank" >https://ieeexplore.ieee.org/abstract/document/9308114</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/WORKS51914.2020.00006" target="_blank" >10.1109/WORKS51914.2020.00006</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Runtime vs Scheduler: Analyzing Dask's Overheads

Popis výsledku v původním jazyce

Dask is a distributed task framework which is commonly used by data scientists to parallelize Python code on computing clusters with little programming effort. It uses a sophisticated work-stealing scheduler which has been hand-tuned to execute task graphs as efficiently as possible. But is scheduler optimization a worthwhile effort for Dask? Our paper shows on many real world task graphs that even a completely random scheduler is surprisingly competitive with its built-in scheduler and that the main bottleneck of Dask lies in its runtime overhead. We develop a drop-in replacement for the Dask central server written in Rust which is backwards compatible with existing Dask programs. Thanks to its efficient runtime, our server implementation is able to scale up to larger clusters than Dask and consistently outperforms it on a variety of task graphs, despite the fact that it uses a simpler scheduling algorithm.

Název v anglickém jazyce

Runtime vs Scheduler: Analyzing Dask's Overheads

Popis výsledku anglicky

Dask is a distributed task framework which is commonly used by data scientists to parallelize Python code on computing clusters with little programming effort. It uses a sophisticated work-stealing scheduler which has been hand-tuned to execute task graphs as efficiently as possible. But is scheduler optimization a worthwhile effort for Dask? Our paper shows on many real world task graphs that even a completely random scheduler is surprisingly competitive with its built-in scheduler and that the main bottleneck of Dask lies in its runtime overhead. We develop a drop-in replacement for the Dask central server written in Rust which is backwards compatible with existing Dask programs. Thanks to its efficient runtime, our server implementation is able to scale up to larger clusters than Dask and consistently outperforms it on a variety of task graphs, despite the fact that it uses a simpler scheduling algorithm.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2020

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of 15th Workshop on Workflows in Support of Large-Scale Science, WORKS 2020 : November 2020

ISBN

978-0-7381-1040-0

ISSN

—

e-ISSN

—

Počet stran výsledku

8

Strana od-do

1-8

Název nakladatele

IEEE - Institute of Electrical and Electronics Engineers

Místo vydání

Piscataway

Místo konání akce

Atlanta

Datum konání akce

11. 11. 2020

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—