Concept-aware Data Construction Improves In-context Learning of Language Models

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14330%2F24%3A00136925" target="_blank" >RIV/00216224:14330/24:00136925 - isvavai.cz</a>

Result on the web

<a href="https://aclanthology.org/2024.findings-acl.733/" target="_blank" >https://aclanthology.org/2024.findings-acl.733/</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Concept-aware Data Construction Improves In-context Learning of Language Models

Original language description

Many recent language models (LMs) are capable of in-context learning (ICL), manifested in the LMs’ ability to perform a new task solely from natural-language instruction. Previous work curating in-context learners assumes that ICL emerges from a vast over-parametrization or the scale of multi-task training. However, recent theoretical work attributes the ICL ability to concept-dependent training data and creates functional in-context learners even in small-scale, synthetic settings.In this work, we practically explore this newly identified axis of ICL quality. We propose Concept-aware Training (CoAT), a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. We find that by using CoAT, pre-trained transformers can learn to better utilise new latent concepts from demonstrations and that such ability makes ICL more robust to the functional deficiencies of the previous models. Finally, we show that concept-aware in-context learners are much more effective in in-context learning a majority of unseen tasks compared to traditional instruction tuning, and fare comparably also to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Findings of the Association for Computational Linguistics ACL 2024

ISBN

9798891760998

ISSN

—

e-ISSN

—

Number of pages

18

Pages from-to

12335-12352

Publisher name

Association for Computational Linguistics

Place of publication

Bangkok, Thailand

Event location

Bangkok, Thailand

Event date

Aug 9, 2024

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

001391786804003