FLAT: Fusing layer representations for more efficient transfer learning in NLP

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A43DLRMEI" target="_blank" >RIV/00216208:11320/25:43DLRMEI - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85201454475&doi=10.1016%2fj.neunet.2024.106631&partnerID=40&md5=59df237eda2be43d2cf0098d7d542033" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85201454475&doi=10.1016%2fj.neunet.2024.106631&partnerID=40&md5=59df237eda2be43d2cf0098d7d542033</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.neunet.2024.106631" target="_blank" >10.1016/j.neunet.2024.106631</a>

Jazyk výsledku

angličtina

Název v původním jazyce

FLAT: Fusing layer representations for more efficient transfer learning in NLP

Popis výsledku v původním jazyce

Parameter efficient transfer learning (PETL) methods provide an efficient alternative for fine-tuning. However, typical PETL methods inject the same structures to all Pre-trained Language Model (PLM) layers and only use the final hidden states for downstream tasks, regardless of the knowledge diversity across PLM layers. Additionally, the backpropagation path of existing PETL methods still passes through the frozen PLM during training, which is computational and memory inefficient. In this paper, we propose FLAT, a generic PETL method that explicitly and individually combines knowledge across all PLM layers based on the tokens to perform a better transferring. FLAT considers the backbone PLM as a feature extractor and combines the features in a side-network, hence the backpropagation does not involve the PLM, which results in much less memory requirement than previous methods. The results on the GLUE benchmark show that FLAT outperforms other tuning techniques in the low-resource scenarios and achieves on-par performance in the high-resource scenarios with only 0.53% trainable parameters per task and 3.2× less GPU memory usagewith BERTbase. Besides, further ablation study is conducted to reveal that the proposed fusion layer effectively combines knowledge from PLM and helps the classifier to exploit the PLM knowledge to downstream tasks. We will release our code for better reproducibility.

Název v anglickém jazyce

FLAT: Fusing layer representations for more efficient transfer learning in NLP

Popis výsledku anglicky

Parameter efficient transfer learning (PETL) methods provide an efficient alternative for fine-tuning. However, typical PETL methods inject the same structures to all Pre-trained Language Model (PLM) layers and only use the final hidden states for downstream tasks, regardless of the knowledge diversity across PLM layers. Additionally, the backpropagation path of existing PETL methods still passes through the frozen PLM during training, which is computational and memory inefficient. In this paper, we propose FLAT, a generic PETL method that explicitly and individually combines knowledge across all PLM layers based on the tokens to perform a better transferring. FLAT considers the backbone PLM as a feature extractor and combines the features in a side-network, hence the backpropagation does not involve the PLM, which results in much less memory requirement than previous methods. The results on the GLUE benchmark show that FLAT outperforms other tuning techniques in the low-resource scenarios and achieves on-par performance in the high-resource scenarios with only 0.53% trainable parameters per task and 3.2× less GPU memory usagewith BERTbase. Besides, further ablation study is conducted to reveal that the proposed fusion layer effectively combines knowledge from PLM and helps the classifier to exploit the PLM knowledge to downstream tasks. We will release our code for better reproducibility.

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Neural Networks

ISSN

0893-6080

e-ISSN

—

Svazek periodika

179

Číslo periodika v rámci svazku

2024

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

15

Strana od-do

1-15

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-85201454475