OpenWebSearch.eu - Building an Open Web Index on EuroHPC JU Infrastructures

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989100%3A27740%2F25%3A10260268" target="_blank" >RIV/61989100:27740/25:10260268 - isvavai.cz</a>

Result on the web

<a href="https://www.sciencedirect.com/science/article/pii/S1877050925006209?via%3Dihub" target="_blank" >https://www.sciencedirect.com/science/article/pii/S1877050925006209?via%3Dihub</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.procs.2025.02.259" target="_blank" >10.1016/j.procs.2025.02.259</a>

Result language

angličtina

Original language name

OpenWebSearch.eu - Building an Open Web Index on EuroHPC JU Infrastructures

Original language description

The OpenWebSearch.eu project aims to develop an Open Web Index (OWI), an openly accessible data structure that supports the creation of web search engines. Building such an index requires a data- and compute-intensive pipeline for cleaning, preprocessing, enriching and indexing large amounts of web data. Beyond search, the availability of clean and preprocessed web data is also crucial for fields like web analytics and generative AI. This paper presents our approach to constructing the OWI using High-Performance Computing (HPC) resources from both EuroHPC JU and non-EuroHPC JU data centers. We contribute in two main areas: first, by detailing the development of pre-processing and indexing pipelines embedded in HPC workflows; and second, by describing the iRODS-based federated storage infrastructure and the LEXIS1 platform that will manage the cross-data centre workflows and facilitate the publication of the OWI as daily datasets. During the alpha phase, from October 2023 to April 2024, we processed approximately 76 TB of web data, encompassing over 2 billion URLs. By addressing the challenges of large-scale web data processing and retrieval, this work lays the foundation for an innovative, competitive, and transparent web search ecosystem, while also supporting the development of European generative AI solutions. © 2025 The Authors. Published by Elsevier B.V.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Procedia Computer Science. Volume 255

ISBN

—

ISSN

1877-0509

e-ISSN

1877-0509

Number of pages

10

Pages from-to

43-52

Publisher name

Elsevier

Place of publication

Amsterdam

Event location

Amsterdam

Event date

Oct 22, 2024

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—