Vše

Co hledáte?

Vše
Projekty
Výsledky výzkumu
Subjekty

Rychlé hledání

  • Projekty podpořené TA ČR
  • Významné projekty
  • Projekty s nejvyšší státní podporou
  • Aktuálně běžící projekty

Chytré vyhledávání

  • Takto najdu konkrétní +slovo
  • Takto z výsledků -slovo zcela vynechám
  • “Takto můžu najít celou frázi”

Task Prompt Vectors: Effective Initialization through Multi-Task Soft Prompt Transfer

Identifikátory výsledku

  • Kód výsledku v IS VaVaI

    <a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26230%2F26%3A0198002" target="_blank" >RIV/00216305:26230/26:0198002 - isvavai.cz</a>

  • Výsledek na webu

    <a href="https://link.springer.com/chapter/10.1007/978-3-662-72243-5_5" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-662-72243-5_5</a>

  • DOI - Digital Object Identifier

    <a href="http://dx.doi.org/10.1007/978-3-662-72243-5_5" target="_blank" >10.1007/978-3-662-72243-5_5</a>

Alternativní jazyky

  • Jazyk výsledku

    angličtina

  • Název v původním jazyce

    Task Prompt Vectors: Effective Initialization through Multi-Task Soft Prompt Transfer

  • Popis výsledku v původním jazyce

    Prompt tuning is a parameter-efficient method for adapting large language models (LLMs), where only a small continuous soft prompt is finetuned. In recent works, soft prompts have usually been trained in a task-specific way, leaving their multi-task capabilities underexplored. Our work aims to make soft prompts more task modular based on recent research on task vectors, where arithmetic operations are applied on full model weights to achieve the desired multi-task performance. To this end, we introduce Task Prompt Vectors, created by the element-wise difference between weights of tuned soft prompts and their random initialization. Experimental results on an extensive set of 19 datasets show that task prompt vectors can be used in low-resource settings to initialize prompt tuning on similar tasks effectively. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning on 3 different language model architectures. This key property of random initialization independence allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, the arithmetic addition of task prompt vectors from multiple tasks represents a competitive and computationally more effective alternative to state-of-the-art solutions.

  • Název v anglickém jazyce

    Task Prompt Vectors: Effective Initialization through Multi-Task Soft Prompt Transfer

  • Popis výsledku anglicky

    Prompt tuning is a parameter-efficient method for adapting large language models (LLMs), where only a small continuous soft prompt is finetuned. In recent works, soft prompts have usually been trained in a task-specific way, leaving their multi-task capabilities underexplored. Our work aims to make soft prompts more task modular based on recent research on task vectors, where arithmetic operations are applied on full model weights to achieve the desired multi-task performance. To this end, we introduce Task Prompt Vectors, created by the element-wise difference between weights of tuned soft prompts and their random initialization. Experimental results on an extensive set of 19 datasets show that task prompt vectors can be used in low-resource settings to initialize prompt tuning on similar tasks effectively. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning on 3 different language model architectures. This key property of random initialization independence allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, the arithmetic addition of task prompt vectors from multiple tasks represents a competitive and computationally more effective alternative to state-of-the-art solutions.

Klasifikace

  • Druh

    O - Ostatní výsledky

  • CEP obor

    —

  • OECD FORD obor

    10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Návaznosti výsledku

  • Projekt

    —

  • Návaznosti

    I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Ostatní

  • Rok uplatnění

    2025

  • Kód důvěrnosti údajů

    S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů