Hierarchical Semi-Sparse Cubes - parallel framework for storing multi-modal big data in HDF5

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21240%2F23%3A00369738" target="_blank" >RIV/68407700:21240/23:00369738 - isvavai.cz</a>

Alternative codes found

RIV/67985815:_____/23:00581668

Result on the web

<a href="https://doi.org/10.1109/ACCESS.2023.3323897" target="_blank" >https://doi.org/10.1109/ACCESS.2023.3323897</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/ACCESS.2023.3323897" target="_blank" >10.1109/ACCESS.2023.3323897</a>

Result language

angličtina

Original language name

Hierarchical Semi-Sparse Cubes - parallel framework for storing multi-modal big data in HDF5

Original language description

Since Moore`s law applies also to data detectors, the volume of data collected in astronomy doubles approximately every year. A prime example is the upcoming Square Kilometer Array (SKA) instrument that will produce approximately 8.5 Exabytes over the first 15 years of service, starting in the year 2027. Storage capacities for these data have grown as well, and primary analytical tools have also kept up. However, the tools for combining big data from several such instruments still lag behind. Having the ability to easily combine big data is crucial for inferring new knowledge about the universe from the correlations and not only finding interesting information in these huge datasets but also their combinations. In this article, we present a revised version of the Hierarchical Semi-Sparse Cube (HiSS-Cube) framework. It aims to provide highly parallel processing of combined multi-modal multi-dimensional big data. The main contributions of this study are as follows: 1) Highly parallel construction of a database built on top of the HDF5 framework. This database supports parallel queries. 2) Design of a database index on top of HDF5 that can be easily constructed in parallel. 3) Support of efficient multi-modal big data combinations. We tested the scalability and efficiency on big astronomical spectroscopic and photometric data obtained from the Sloan Digital Sky Survey. The performance of HiSS-Cube is bounded by the I/O bandwidth and I/O operations per second of the underlying parallel file system, and it scales linearly with the number of I/O nodes.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Research Center for Informatics</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

IEEE Access

ISSN

2169-3536

e-ISSN

2169-3536

Volume of the periodical

11

Issue of the periodical within the volume

October

Country of publishing house

US - UNITED STATES

Number of pages

22

Pages from-to

119876-119897

UT code for WoS article

001100997000001

EID of the result in the Scopus database

2-s2.0-85174830822