A Study of a Cross-modal Interactive Search Tool Using CLIP and Temporal Fusion

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A10469910" target="_blank" >RIV/00216208:11320/23:10469910 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1007/978-3-031-27077-2_31" target="_blank" >https://doi.org/10.1007/978-3-031-27077-2_31</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-031-27077-2_31" target="_blank" >10.1007/978-3-031-27077-2_31</a>

Jazyk výsledku

angličtina

Název v původním jazyce

A Study of a Cross-modal Interactive Search Tool Using CLIP and Temporal Fusion

Popis výsledku v původním jazyce

Recently, the CLIP model demonstrated impressive performance in text-image search and zero classification tasks. Hence, CLIP was used as a primary model in many cross-modal search tools at evaluation campaigns. In this paper, we show a study performed with the model integrated to a successful video search tool at the respected Video Browser Showdown competition. The tool allows more complex querying actions on top of the primarymodel. Specifically, temporal querying and Bayesian like relevance feedback were tested as well as their natural combination temporal relevance feedback. In a thorough analysis of the tool&apos;s performance, we show current limits of cross-modal searching with CLIP and also the impact of more advanced query formulation strategies. We conclude that current cross-modal search models enable users to solve some types of tasks trivially with a single query, however, for more challenging tasks it is necessary to rely also on interactive search strategies.

Název v anglickém jazyce

A Study of a Cross-modal Interactive Search Tool Using CLIP and Temporal Fusion

Popis výsledku anglicky

Recently, the CLIP model demonstrated impressive performance in text-image search and zero classification tasks. Hence, CLIP was used as a primary model in many cross-modal search tools at evaluation campaigns. In this paper, we show a study performed with the model integrated to a successful video search tool at the respected Video Browser Showdown competition. The tool allows more complex querying actions on top of the primarymodel. Specifically, temporal querying and Bayesian like relevance feedback were tested as well as their natural combination temporal relevance feedback. In a thorough analysis of the tool&apos;s performance, we show current limits of cross-modal searching with CLIP and also the impact of more advanced query formulation strategies. We conclude that current cross-modal search models enable users to solve some types of tasks trivially with a single query, however, for more challenging tasks it is necessary to rely also on interactive search strategies.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GA22-21696S" target="_blank" >GA22-21696S: Hluboké vizuální reprezentace nestrukturovaných dat</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

MULTIMEDIA MODELING, MMM 2023, PT I

ISBN

978-3-031-27076-5

ISSN

0302-9743

e-ISSN

1611-3349

Počet stran výsledku

12

Strana od-do

397-408

Název nakladatele

SPRINGER INTERNATIONAL PUBLISHING AG

Místo vydání

CHAM

Místo konání akce

Bergen

Datum konání akce

9. 1. 2023

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

000996563000031