All

What are you looking for?

All
Projects
Results
Organizations

Quick search

  • Projects supported by TA ČR
  • Excellent projects
  • Projects with the highest public support
  • Current projects

Smart search

  • That is how I find a specific +word
  • That is how I leave the -word out of the results
  • “That is how I can find the whole phrase”

Language-Guided Music Recommendation for Video via Prompt Analogies

The result's identifiers

  • Result code in IS VaVaI

    <a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21730%2F23%3A00372044" target="_blank" >RIV/68407700:21730/23:00372044 - isvavai.cz</a>

  • Result on the web

    <a href="https://doi.org/10.1109/CVPR52729.2023.01420" target="_blank" >https://doi.org/10.1109/CVPR52729.2023.01420</a>

  • DOI - Digital Object Identifier

    <a href="http://dx.doi.org/10.1109/CVPR52729.2023.01420" target="_blank" >10.1109/CVPR52729.2023.01420</a>

Alternative languages

  • Result language

    angličtina

  • Original language name

    Language-Guided Music Recommendation for Video via Prompt Analogies

  • Original language description

    We propose a method to recommend music for an input video while allowing a user to guide music selection with free-form natural language. A key challenge of this problem setting is that existing music video datasets provide the needed (video, music) training pairs, but lack text descriptions of the music. This work addresses this challenge with the following three contributions. First, we propose a text-synthesis approach that relies on an analogy-based prompting procedure to generate natural language music descriptions from a large-scale language model (BLOOM-176B) given pre-trained music tagger outputs and a small number of human text descriptions. Second, we use these synthesized music descriptions to train a new trimodal model, which fuses text and video input representations to query music samples. For training, we introduce a text dropout regularization mechanism which we show is critical to model performance. Our model design allows for the re-trieved music audio to agree with the two input modalities by matching visual style depicted in the video and musical genre, mood, or instrumentation described in the natural language query. Third, to evaluate our approach, we collect a testing dataset for our problem by annotating a subset of 4k clips from the YT8M-Music Video dataset with natural language music descriptions which we make publicly available. We show that our approach can match or exceed the performance of prior methods on video-to-music retrieval while significantly improving retrieval accuracy when using text guidance.

  • Czech name

  • Czech description

Classification

  • Type

    D - Article in proceedings

  • CEP classification

  • OECD FORD branch

    10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Result continuities

  • Project

  • Continuities

    I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Others

  • Publication year

    2023

  • Confidentiality

    S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Data specific for result type

  • Article name in the collection

    Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  • ISBN

    979-8-3503-0130-4

  • ISSN

    1063-6919

  • e-ISSN

    2575-7075

  • Number of pages

    10

  • Pages from-to

    14784-14793

  • Publisher name

    IEEE Computer Society

  • Place of publication

    USA

  • Event location

    Vancouver

  • Event date

    Jun 18, 2023

  • Type of event by nationality

    WRD - Celosvětová akce

  • UT code for WoS article

    001062522107011