Speeding up publication of Linked Data using data chunking in LinkedPipes ETL
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F17%3A10363612" target="_blank" >RIV/00216208:11320/17:10363612 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1007/978-3-319-69459-7_10" target="_blank" >https://doi.org/10.1007/978-3-319-69459-7_10</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/978-3-319-69459-7_10" target="_blank" >10.1007/978-3-319-69459-7_10</a>
Alternative languages
Result language
angličtina
Original language name
Speeding up publication of Linked Data using data chunking in LinkedPipes ETL
Original language description
There is a multitude of tools for preparation of Linked Data from data sources such as CSV and XML files. These tools usually perform as expected when processing examples, or smaller real world data. However, a majority of these tools become hard to use when faced with a larger dataset such as hundreds of megabytes large CSV file. Tools which load the entire resulting RDF dataset into memory usually have memory requirements unsatisfiable by commodity hardware. This is the case of RDF-based ETL tools. Their limits can be avoided by running them on powerful and expensive hardware, which is, however, not an option for majority of data publishers. Tools which process the data in a streamed way tend to have limited transformation options. This is the case of text-based transformations, such as XSLT, or per-item SPARQL transformations such as the streamed version of TARQL. In this paper, we show how the power and transformation options of RDF-based ETL tools can be combined with the possibility to transform large datasets on common consumer hardware for so called chunkable data - data which can be split in a certain way. We demonstrate our approach in our RDF-based ETL tool, LinkedPipes ETL. We include experiments on selected real world datasets and a comparison of performance and memory consumption of available tools.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/GA16-09713S" target="_blank" >GA16-09713S: Efficient Exploration of Linked Data Cloud</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2017
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Confederated International Conferences: CoopIS, C&TC, and ODBASE 2017, Rhodes, Greece, October 23-27, 2017, Proceedings, Part II
ISBN
978-3-319-69459-7
ISSN
0302-9743
e-ISSN
neuvedeno
Number of pages
17
Pages from-to
144-160
Publisher name
Springer
Place of publication
Berlin
Event location
Rhodes, Greece
Event date
Oct 23, 2017
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—