DUAL PROCESS LEARNING: CONTROLLING THE USE OF IN-CONTEXT VS. IN-WEIGHTS STRATEGIES WITH WEIGHT FORGETTING

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AWGDARPYG" target="_blank" >RIV/00216208:11320/26:WGDARPYG - isvavai.cz</a>

Výsledek na webu

<a href="https://www.semanticscholar.org/reader/4211000a615716f4e2109efa5a9416102e3c2afa" target="_blank" >https://www.semanticscholar.org/reader/4211000a615716f4e2109efa5a9416102e3c2afa</a>

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

DUAL PROCESS LEARNING: CONTROLLING THE USE OF IN-CONTEXT VS. IN-WEIGHTS STRATEGIES WITH WEIGHT FORGETTING

Popis výsledku v původním jazyce

Language models have the ability to perform in-context learning (ICL), allowing them to flexibly adapt their behavior based on context. This contrasts with in-weights learning (IWL), where memorized information is encoded in model parameters after iterated observations of data. An ideal model should be able to flexibly deploy both of these abilities. Despite their apparent ability to learn in-context, language models are known to struggle when faced with unseen or rarely seen tokens (Land & Bartolo, 2024). Hence, we study structural in-context learning, which we define as the ability of a model to execute in-context learning on arbitrary novel tokens - so called because the model must generalize on the basis of e.g. sentence structure or task structure, rather than content encoded in token embeddings. We study structural in-context algorithms on both synthetic and naturalistic tasks using toy models, masked language models, and autoregressive language models. We find that structural ICL appears before quickly disappearing early in LM pretraining. While it has been shown that ICL can diminish during training (Singh et al., 2023), we find that prior work does not account for structural ICL. Building on Chen et al. (2024)'s active forgetting method, we introduce pretraining and finetuning methods that can modulate the preference for structural ICL and IWL. Importantly, this allows us to induce a dual process strategy where in-context and in-weights solutions coexist within a single model. © 2025 13th International Conference on Learning Representations, ICLR 2025. All rights reserved.

Název v anglickém jazyce

DUAL PROCESS LEARNING: CONTROLLING THE USE OF IN-CONTEXT VS. IN-WEIGHTS STRATEGIES WITH WEIGHT FORGETTING

Popis výsledku anglicky

Language models have the ability to perform in-context learning (ICL), allowing them to flexibly adapt their behavior based on context. This contrasts with in-weights learning (IWL), where memorized information is encoded in model parameters after iterated observations of data. An ideal model should be able to flexibly deploy both of these abilities. Despite their apparent ability to learn in-context, language models are known to struggle when faced with unseen or rarely seen tokens (Land & Bartolo, 2024). Hence, we study structural in-context learning, which we define as the ability of a model to execute in-context learning on arbitrary novel tokens - so called because the model must generalize on the basis of e.g. sentence structure or task structure, rather than content encoded in token embeddings. We study structural in-context algorithms on both synthetic and naturalistic tasks using toy models, masked language models, and autoregressive language models. We find that structural ICL appears before quickly disappearing early in LM pretraining. While it has been shown that ICL can diminish during training (Singh et al., 2023), we find that prior work does not account for structural ICL. Building on Chen et al. (2024)'s active forgetting method, we introduce pretraining and finetuning methods that can modulate the preference for structural ICL and IWL. Importantly, this allows us to induce a dual process strategy where in-context and in-weights solutions coexist within a single model. © 2025 13th International Conference on Learning Representations, ICLR 2025. All rights reserved.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Int. Conf. Learn. Represent., ICLR

ISBN

979-8-3313-2085-0

ISSN

—

e-ISSN

—

Počet stran výsledku

27

Strana od-do

48442-48468

Název nakladatele

International Conference on Learning Representations, ICLR

Místo vydání

—

Místo konání akce

Singapore

Datum konání akce

1. 1. 2026

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—