InA: Inhibition Adaption on pre-trained language models

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F24%3A00377574" target="_blank" >RIV/68407700:21230/24:00377574 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1016/j.neunet.2024.106410" target="_blank" >https://doi.org/10.1016/j.neunet.2024.106410</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.neunet.2024.106410" target="_blank" >10.1016/j.neunet.2024.106410</a>

Jazyk výsledku

angličtina

Název v původním jazyce

InA: Inhibition Adaption on pre-trained language models

Popis výsledku v původním jazyce

Fine-tuning pre-trained language models (LMs) may not always be the most practical approach for downstream tasks. While adaptation fine-tuning methods have shown promising results, a clearer explanation of their mechanisms and further inhibition of the transmission of information is needed. To address this, we propose an Inhibition Adaptation (InA) fine-tuning method that aims to reduce the number of added tunable weights and appropriately reweight knowledge derived from pre-trained LMs. The InA method involves (1) inserting a small trainable vector into each Transformer attention architecture and (2) setting a threshold to directly eliminate irrelevant knowledge. This approach draws inspiration from the shunting inhibition, which allows the inhibition of specific neurons to gate other functional neurons. With the inhibition mechanism, InA achieves competitive or even superior performance compared to other fine-tuning methods on ???????????????? - ????????????????????, ???????????????????????? ???? - ????????????????????, and ???????????????????????? ???? - ???????????????????? for text classification and question-answering tasks.

Název v anglickém jazyce

InA: Inhibition Adaption on pre-trained language models

Popis výsledku anglicky

Fine-tuning pre-trained language models (LMs) may not always be the most practical approach for downstream tasks. While adaptation fine-tuning methods have shown promising results, a clearer explanation of their mechanisms and further inhibition of the transmission of information is needed. To address this, we propose an Inhibition Adaptation (InA) fine-tuning method that aims to reduce the number of added tunable weights and appropriately reweight knowledge derived from pre-trained LMs. The InA method involves (1) inserting a small trainable vector into each Transformer attention architecture and (2) setting a threshold to directly eliminate irrelevant knowledge. This approach draws inspiration from the shunting inhibition, which allows the inhibition of specific neurons to gate other functional neurons. With the inhibition mechanism, InA achieves competitive or even superior performance compared to other fine-tuning methods on ???????????????? - ????????????????????, ???????????????????????? ???? - ????????????????????, and ???????????????????????? ???? - ???????????????????? for text classification and question-answering tasks.

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Výzkumné centrum informatiky</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Neural Networks

ISSN

0893-6080

e-ISSN

1879-2782

Svazek periodika

178

Číslo periodika v rámci svazku

Oct

Stát vydavatele periodika

NL - Nizozemsko

Počet stran výsledku

12

Strana od-do

—

Kód UT WoS článku

001251421000001

EID výsledku v databázi Scopus

2-s2.0-85195198608