Speeding up publication of Linked Data using data chunking in LinkedPipes ETL

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F17%3A10363612" target="_blank" >RIV/00216208:11320/17:10363612 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1007/978-3-319-69459-7_10" target="_blank" >https://doi.org/10.1007/978-3-319-69459-7_10</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-319-69459-7_10" target="_blank" >10.1007/978-3-319-69459-7_10</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Speeding up publication of Linked Data using data chunking in LinkedPipes ETL

Popis výsledku v původním jazyce

There is a multitude of tools for preparation of Linked Data from data sources such as CSV and XML files. These tools usually perform as expected when processing examples, or smaller real world data. However, a majority of these tools become hard to use when faced with a larger dataset such as hundreds of megabytes large CSV file. Tools which load the entire resulting RDF dataset into memory usually have memory requirements unsatisfiable by commodity hardware. This is the case of RDF-based ETL tools. Their limits can be avoided by running them on powerful and expensive hardware, which is, however, not an option for majority of data publishers. Tools which process the data in a streamed way tend to have limited transformation options. This is the case of text-based transformations, such as XSLT, or per-item SPARQL transformations such as the streamed version of TARQL. In this paper, we show how the power and transformation options of RDF-based ETL tools can be combined with the possibility to transform large datasets on common consumer hardware for so called chunkable data - data which can be split in a certain way. We demonstrate our approach in our RDF-based ETL tool, LinkedPipes ETL. We include experiments on selected real world datasets and a comparison of performance and memory consumption of available tools.

Název v anglickém jazyce

Speeding up publication of Linked Data using data chunking in LinkedPipes ETL

Popis výsledku anglicky

There is a multitude of tools for preparation of Linked Data from data sources such as CSV and XML files. These tools usually perform as expected when processing examples, or smaller real world data. However, a majority of these tools become hard to use when faced with a larger dataset such as hundreds of megabytes large CSV file. Tools which load the entire resulting RDF dataset into memory usually have memory requirements unsatisfiable by commodity hardware. This is the case of RDF-based ETL tools. Their limits can be avoided by running them on powerful and expensive hardware, which is, however, not an option for majority of data publishers. Tools which process the data in a streamed way tend to have limited transformation options. This is the case of text-based transformations, such as XSLT, or per-item SPARQL transformations such as the streamed version of TARQL. In this paper, we show how the power and transformation options of RDF-based ETL tools can be combined with the possibility to transform large datasets on common consumer hardware for so called chunkable data - data which can be split in a certain way. We demonstrate our approach in our RDF-based ETL tool, LinkedPipes ETL. We include experiments on selected real world datasets and a comparison of performance and memory consumption of available tools.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GA16-09713S" target="_blank" >GA16-09713S: Efektivní explorace prostoru propojených dat</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2017

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Confederated International Conferences: CoopIS, C&amp;TC, and ODBASE 2017, Rhodes, Greece, October 23-27, 2017, Proceedings, Part II

ISBN

978-3-319-69459-7

ISSN

0302-9743

e-ISSN

neuvedeno

Počet stran výsledku

17

Strana od-do

144-160

Název nakladatele

Springer

Místo vydání

Berlin

Místo konání akce

Rhodes, Greece

Datum konání akce

23. 10. 2017

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—