Task Prompt Vectors: Effective Initialization through Multi-Task Soft Prompt Transfer
Identifikátory výsledku
Kód výsledku v IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26230%2F26%3A0198002" target="_blank" >RIV/00216305:26230/26:0198002 - isvavai.cz</a>
Výsledek na webu
<a href="https://link.springer.com/chapter/10.1007/978-3-662-72243-5_5" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-662-72243-5_5</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/978-3-662-72243-5_5" target="_blank" >10.1007/978-3-662-72243-5_5</a>
Alternativní jazyky
Jazyk výsledku
angličtina
Název v původním jazyce
Task Prompt Vectors: Effective Initialization through Multi-Task Soft Prompt Transfer
Popis výsledku v původním jazyce
Prompt tuning is a parameter-efficient method for adapting large language models (LLMs), where only a small continuous soft prompt is finetuned. In recent works, soft prompts have usually been trained in a task-specific way, leaving their multi-task capabilities underexplored. Our work aims to make soft prompts more task modular based on recent research on task vectors, where arithmetic operations are applied on full model weights to achieve the desired multi-task performance. To this end, we introduce Task Prompt Vectors, created by the element-wise difference between weights of tuned soft prompts and their random initialization. Experimental results on an extensive set of 19 datasets show that task prompt vectors can be used in low-resource settings to initialize prompt tuning on similar tasks effectively. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning on 3 different language model architectures. This key property of random initialization independence allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, the arithmetic addition of task prompt vectors from multiple tasks represents a competitive and computationally more effective alternative to state-of-the-art solutions.
Název v anglickém jazyce
Task Prompt Vectors: Effective Initialization through Multi-Task Soft Prompt Transfer
Popis výsledku anglicky
Prompt tuning is a parameter-efficient method for adapting large language models (LLMs), where only a small continuous soft prompt is finetuned. In recent works, soft prompts have usually been trained in a task-specific way, leaving their multi-task capabilities underexplored. Our work aims to make soft prompts more task modular based on recent research on task vectors, where arithmetic operations are applied on full model weights to achieve the desired multi-task performance. To this end, we introduce Task Prompt Vectors, created by the element-wise difference between weights of tuned soft prompts and their random initialization. Experimental results on an extensive set of 19 datasets show that task prompt vectors can be used in low-resource settings to initialize prompt tuning on similar tasks effectively. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning on 3 different language model architectures. This key property of random initialization independence allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, the arithmetic addition of task prompt vectors from multiple tasks represents a competitive and computationally more effective alternative to state-of-the-art solutions.
Klasifikace
Druh
O - Ostatní výsledky
CEP obor
—
OECD FORD obor
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Návaznosti výsledku
Projekt
—
Návaznosti
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Ostatní
Rok uplatnění
2025
Kód důvěrnosti údajů
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů