Efficient handling of lots of simulation data files

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216275%3A25530%2F22%3A39919497" target="_blank" >RIV/00216275:25530/22:39919497 - isvavai.cz</a>

Result on the web

<a href="http://dx.doi.org/10.46354/i3m.2022.emss.043" target="_blank" >http://dx.doi.org/10.46354/i3m.2022.emss.043</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.46354/i3m.2022.emss.043" target="_blank" >10.46354/i3m.2022.emss.043</a>

Result language

angličtina

Original language name

Efficient handling of lots of simulation data files

Original language description

Saving information to files is the most basic and simplest way to store data, so it is often used in simple simulators and simulation tools as the first choice for logging information about the simulation process and its results. Computer simulations often involve simulating a significant number of replications and accumulating large numbers of files. Today&apos;s filesystems are still not capable of efficiently storing and processing millions of files. This paper presents alternatives that allow for more efficient storage, transfer, and analysis of data, with an emphasis on easy migration or implementation from the initial data files. Simple approaches such as using TAR or ZIP archives to sophisticated approaches involving Parquet file, S3-like object storage (e.g., MinIO, OpenIO) and (non)relational databases will be compared.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach

Publication year

2022

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the 34th European Modeling &amp; Simulation Symposium (EMSS 2022)

ISBN

978-88-85741-73-7

ISSN

2724-0029

e-ISSN

2724-0029

Number of pages

4

Pages from-to

1-4

Publisher name

CAL-TEK SRL

Place of publication

Rende

Event location

Řím

Event date

Sep 19, 2022

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—