WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F24%3A10492883" target="_blank" >RIV/00216208:11320/24:10492883 - isvavai.cz</a>

Výsledek na webu

<a href="https://openreview.net/forum?id=mUSPhG4uDW" target="_blank" >https://openreview.net/forum?id=mUSPhG4uDW</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.48550/arXiv.2402.05930" target="_blank" >10.48550/arXiv.2402.05930</a>

Jazyk výsledku

angličtina

Název v původním jazyce

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Popis výsledku v původním jazyce

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve realworld tasks in a multi-turn dialogue fashion. To support this problem, we introduce WebLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers abroad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller fi

Název v anglickém jazyce

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

Popis výsledku anglicky

We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve realworld tasks in a multi-turn dialogue fashion. To support this problem, we introduce WebLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers abroad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller fi

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

R - Projekt Ramcoveho programu EK

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 41st International Conference on Machine Learning

ISBN

979-8-3313-0223-8

ISSN

2640-3498

e-ISSN

—

Počet stran výsledku

50

Strana od-do

1-50

Název nakladatele

Proceedings of Machine Learning Research (PMLR)

Místo vydání

San Diego, USA

Místo konání akce

Wien, Austria

Datum konání akce

21. 7. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—