Offline Environment for Focused Crawler Evaluation

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F16%3A00243403" target="_blank" >RIV/68407700:21230/16:00243403 - isvavai.cz</a>

Výsledek na webu

—

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Offline Environment for Focused Crawler Evaluation

Popis výsledku v původním jazyce

Focused crawling is a method for automatic exploration of the World Wide Web focusing on regions, which are relevant to a given topic. Since focused crawlers are often used to collect data for vertical search engines, they received special attention in research community. Unfortunately, dynamic nature of the internet complicates fair comparisons of focused crawlers, because crawling experiments are not repeatable in the longer term. Researchers try to mitigate the impact of a changing environment by running tested crawlers simultaneously. However, this requirement makes extensive comparisons almost impossible. In this work we try to solve this issue by proposing a system for offline evaluation, which allows developers to consistently test their algorithms on fixed subgraph of the internet. This subgraph consists of 3.5 billion web pages and 128.7 billion links. Our experiments with baseline focused crawlers showed that the performance achieved in the offline environment correlates with the performance achieved online. This suggests that the proposed environment, which allows us to carry out repeatable experiments, can be used for focused crawler comparisons.

Název v anglickém jazyce

Offline Environment for Focused Crawler Evaluation

Popis výsledku anglicky

Focused crawling is a method for automatic exploration of the World Wide Web focusing on regions, which are relevant to a given topic. Since focused crawlers are often used to collect data for vertical search engines, they received special attention in research community. Unfortunately, dynamic nature of the internet complicates fair comparisons of focused crawlers, because crawling experiments are not repeatable in the longer term. Researchers try to mitigate the impact of a changing environment by running tested crawlers simultaneously. However, this requirement makes extensive comparisons almost impossible. In this work we try to solve this issue by proposing a system for offline evaluation, which allows developers to consistently test their algorithms on fixed subgraph of the internet. This subgraph consists of 3.5 billion web pages and 128.7 billion links. Our experiments with baseline focused crawlers showed that the performance achieved in the offline environment correlates with the performance achieved online. This suggests that the proposed environment, which allows us to carry out repeatable experiments, can be used for focused crawler comparisons.

Druh

D - Stať ve sborníku

CEP obor

JC - Počítačový hardware a software

OECD FORD obor

—

Projekt

—

Návaznosti

S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2016

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 20th International Scientific Student Conferenece POSTER 2016

ISBN

978-80-01-05950-0

ISSN

—

e-ISSN

—

Počet stran výsledku

8

Strana od-do

—

Název nakladatele

Czech Technical University in Prague

Místo vydání

Praha

Místo konání akce

Praha

Datum konání akce

24. 5. 2016

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—