InA: Inhibition Adaption on pre-trained language models

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F24%3A00377574" target="_blank" >RIV/68407700:21230/24:00377574 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1016/j.neunet.2024.106410" target="_blank" >https://doi.org/10.1016/j.neunet.2024.106410</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1016/j.neunet.2024.106410" target="_blank" >10.1016/j.neunet.2024.106410</a>

Result language
angličtina
Original language name
InA: Inhibition Adaption on pre-trained language models
Original language description
Fine-tuning pre-trained language models (LMs) may not always be the most practical approach for downstream tasks. While adaptation fine-tuning methods have shown promising results, a clearer explanation of their mechanisms and further inhibition of the transmission of information is needed. To address this, we propose an Inhibition Adaptation (InA) fine-tuning method that aims to reduce the number of added tunable weights and appropriately reweight knowledge derived from pre-trained LMs. The InA method involves (1) inserting a small trainable vector into each Transformer attention architecture and (2) setting a threshold to directly eliminate irrelevant knowledge. This approach draws inspiration from the shunting inhibition, which allows the inhibition of specific neurons to gate other functional neurons. With the inhibition mechanism, InA achieves competitive or even superior performance compared to other fine-tuning methods on ???????????????? - ????????????????????, ???????????????????????? ???? - ????????????????????, and ???????????????????????? ???? - ???????????????????? for text classification and question-answering tasks.
Czech name
—
Czech description
—

Type
J<sub>imp</sub> - Article in a specialist periodical, which is included in the Web of Science database
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project
<a href="/en/project/EF16_019%2F0000765" target="_blank" >EF16_019/0000765: Research Center for Informatics</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Similar results(10)