Scalable Similarity Joins for Fast and Accurate Record Deduplication in Big Data

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216275%3A25530%2F24%3A39921108" target="_blank" >RIV/00216275:25530/24:39921108 - isvavai.cz</a>

Výsledek na webu

<a href="https://link.springer.com/chapter/10.1007/978-3-031-60328-0_18" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-031-60328-0_18</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-031-60328-0_18" target="_blank" >10.1007/978-3-031-60328-0_18</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Scalable Similarity Joins for Fast and Accurate Record Deduplication in Big Data

Popis výsledku v původním jazyce

Record linkage is the process of matching records from multiple data sources that refer to the same entities. When applied to a single data source, this process is known as deduplication. With the increasing size of data source, recently referred to as big data, the complexity of the matching process becomes one of the major challenges for record linkage and deduplication. In recent decades, several blocking, indexing and filtering techniques have been developed. Their purpose is to reduce the number of record pairs to be compared by removing obvious non-matching pairs in the deduplication process, while maintaining high quality of matching. Currently developed algorithms and traditional techniques are not efficient, using methods that still lose significant proportion of true matches when removing comparison pairs. This paper proposes more efficient algorithms for removing non-matching pairs, with an explicitly proven mathematical lower bound on recently used stateof-the-art approximate string matching method - Fuzzy Jaccard Similarity. The algorithm is also much more efficient in classification using Density-based spatial clustering of applications with noise (DBSCAN) in log-linear time complexity O(|E| log(|E|)).

Název v anglickém jazyce

Scalable Similarity Joins for Fast and Accurate Record Deduplication in Big Data

Popis výsledku anglicky

Record linkage is the process of matching records from multiple data sources that refer to the same entities. When applied to a single data source, this process is known as deduplication. With the increasing size of data source, recently referred to as big data, the complexity of the matching process becomes one of the major challenges for record linkage and deduplication. In recent decades, several blocking, indexing and filtering techniques have been developed. Their purpose is to reduce the number of record pairs to be compared by removing obvious non-matching pairs in the deduplication process, while maintaining high quality of matching. Currently developed algorithms and traditional techniques are not efficient, using methods that still lose significant proportion of true matches when removing comparison pairs. This paper proposes more efficient algorithms for removing non-matching pairs, with an explicitly proven mathematical lower bound on recently used stateof-the-art approximate string matching method - Fuzzy Jaccard Similarity. The algorithm is also much more efficient in classification using Density-based spatial clustering of applications with noise (DBSCAN) in log-linear time complexity O(|E| log(|E|)).

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Good Practices and New Perspectives in Information Systems and Technologies : WorldCIST 2024, Volume 6

ISBN

978-3-031-60327-3

ISSN

2367-3370

e-ISSN

2367-3389

Počet stran výsledku

11

Strana od-do

"181 "- 191

Název nakladatele

Springer Nature Switzerland AG

Místo vydání

Cham

Místo konání akce

Lodž

Datum konání akce

26. 3. 2024

Typ akce podle státní příslušnosti

EUR - Evropská akce

Kód UT WoS článku

001267244400018