A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26230%2F24%3APU154587" target="_blank" >RIV/00216305:26230/24:PU154587 - isvavai.cz</a>

Result on the web

<a href="https://dl.acm.org/doi/10.1145/3691339?cid=81474695895" target="_blank" >https://dl.acm.org/doi/10.1145/3691339?cid=81474695895</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3691339" target="_blank" >10.1145/3691339</a>

Result language

angličtina

Original language name

A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness

Original language description

Learning with limited labelled data, such as prompting, in-context learning, fine-tuning, meta-learning or few-shot learning, aims to effectively train a model using only a small amount of labelled samples. However, these approaches have been observed to be excessively sensitive to the effects of uncontrolled randomness caused by non-determinism in the training process. The randomness negatively affects the stability of the models, leading to large variances in results across training runs. When such sensitivity is disregarded, it can unintentionally, but unfortunately also intentionally, create an imaginary perception of research progress. Recently, this area started to attract research attention and the number of relevant studies is continuously growing. In this survey, we provide a comprehensive overview of 415 papers addressing the effects of randomness on the stability of learning with limited labelled data. We distinguish between four main tasks addressed in the papers (investigate/evaluate; determine; mitigate; benchmark/compare/report randomness effects), providing findings for each one. Furthermore, we identify and discuss seven challenges and open problems together with possible directions to facilitate further research. The ultimate goal of this survey is to emphasise the importance of this growing research area, which so far has not received an appropriate level of attention, and reveal impactful directions for future research.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

ACM COMPUTING SURVEYS

ISSN

0360-0300

e-ISSN

1557-7341

Volume of the periodical

57

Issue of the periodical within the volume

1

Country of publishing house

US - UNITED STATES

Number of pages

40

Pages from-to

1-40

UT code for WoS article

001368989600008

EID of the result in the Scopus database

2-s2.0-85208394765