Simulations and study of a new scheduling approach for distributed data production

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61389005%3A_____%2F16%3A00506284" target="_blank" >RIV/61389005:_____/16:00506284 - isvavai.cz</a>

Nalezeny alternativní kódy

RIV/00216224:14330/16:00088810

Výsledek na webu

<a href="http://dx.doi.org/10.1088/1742-6596/762/1/012023" target="_blank" >http://dx.doi.org/10.1088/1742-6596/762/1/012023</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1088/1742-6596/762/1/012023" target="_blank" >10.1088/1742-6596/762/1/012023</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Simulations and study of a new scheduling approach for distributed data production

Popis výsledku v původním jazyce

Distributed data processing has found its application in many fields of science (High Energy and Nuclear Physics (HENP), astronomy, biology to name only those). We have focused our research on distributed data production, an essential part of computations in HENP. Using our previous experience, we have recently proposed a new scheduling approach for distributed data production which is based on the network flow maximization model. It has a polynomial complexity providing required scalability with respect to the size of computations. Our approach improves the overall data production throughput due to three factors: transfer input files in advance before their processing (allows to decrease I/O latency). Balancing of the network traffic (includes splitting the load between several alternative transfer paths), and transfer files sequentially in a coordinated manner (allows to reduce the influence of possible network bottlenecks). In this contribution, we present the results of our new simulations based on the GridSim framework which is one of the commonly used tools in the field of distributed computations. In these simulations we study the behavior of standard scheduling approaches compared to our recently proposed approach in a realistic environment relying on the data from the STAR and ATLAS experiments and considering the influence of the background traffic. The final goal of the research is to integrate the proposed scheduling approach into the real data production framework. In order to achieve this we are constantly moving our simulations towards real use cases, study scalability of the model and the influence of the scheduling parameters on the quality of the solution.

Název v anglickém jazyce

Simulations and study of a new scheduling approach for distributed data production

Popis výsledku anglicky

Distributed data processing has found its application in many fields of science (High Energy and Nuclear Physics (HENP), astronomy, biology to name only those). We have focused our research on distributed data production, an essential part of computations in HENP. Using our previous experience, we have recently proposed a new scheduling approach for distributed data production which is based on the network flow maximization model. It has a polynomial complexity providing required scalability with respect to the size of computations. Our approach improves the overall data production throughput due to three factors: transfer input files in advance before their processing (allows to decrease I/O latency). Balancing of the network traffic (includes splitting the load between several alternative transfer paths), and transfer files sequentially in a coordinated manner (allows to reduce the influence of possible network bottlenecks). In this contribution, we present the results of our new simulations based on the GridSim framework which is one of the commonly used tools in the field of distributed computations. In these simulations we study the behavior of standard scheduling approaches compared to our recently proposed approach in a realistic environment relying on the data from the STAR and ATLAS experiments and considering the influence of the background traffic. The final goal of the research is to integrate the proposed scheduling approach into the real data production framework. In order to achieve this we are constantly moving our simulations towards real use cases, study scalability of the model and the influence of the scheduling parameters on the quality of the solution.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10304 - Nuclear physics

Projekt

<a href="/cs/project/LG15001" target="_blank" >LG15001: Účast České republiky na experimentech v Brookhavenské národní laboratoři v USA</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2016

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Journal of Physics Conference Series

ISBN

—

ISSN

1742-6588

e-ISSN

—

Počet stran výsledku

7

Strana od-do

012023

Název nakladatele

IOP Publishing

Místo vydání

Bristol

Místo konání akce

Valparaiso

Datum konání akce

18. 1. 2016

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

000439689600023