TopoBERT: Exploring the topology of fine-tuned word representations

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3ADAWIB9WC" target="_blank" >RIV/00216208:11320/23:DAWIB9WC - isvavai.cz</a>

Result on the web

<a href="http://journals.sagepub.com/doi/10.1177/14738716231168671" target="_blank" >http://journals.sagepub.com/doi/10.1177/14738716231168671</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1177/14738716231168671" target="_blank" >10.1177/14738716231168671</a>

Result language

angličtina

Original language name

TopoBERT: Exploring the topology of fine-tuned word representations

Original language description

"Transformer-based language models such as BERT and its variants have found widespread use in natural language processing (NLP). A common way of using these models is to fine-tune them to improve their performance on a specific task. However, it is currently unclear how the fine-tuning process affects the underlying structure of the word embeddings from these models. We present TopoBERT, a visual analytics system for interactively exploring the fine-tuning process of various transformer-based models – across multiple fine-tuning batch updates, subsequent layers of the model, and different NLP tasks – from a topological perspective. The system uses the mapper algorithm from topological data analysis (TDA) to generate a graph that approximates the shape of a model’s embedding space for an input dataset. TopoBERT enables its users (e.g. experts in NLP and linguistics) to (1) interactively explore the fine-tuning process across different model-task pairs, (2) visualize the shape of embedding spaces at multiple scales and layers, and (3) connect linguistic and contextual information about the input dataset with the topology of the embedding space. Using TopoBERT, we provide various use cases to exemplify its applications in exploring fine-tuned word embeddings. We further demonstrate the utility of TopoBERT, which enables users to generate insights about the fine-tuning process and provides support for empirical validation of these insights."

Czech name

—

Czech description

—

Type

J_ost - Miscellaneous article in a specialist periodical

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

"Information Visualization"

ISSN

1473-8716

e-ISSN

—

Volume of the periodical

22

Issue of the periodical within the volume

3

Country of publishing house

US - UNITED STATES

Number of pages

23

Pages from-to

186-208

UT code for WoS article

—

EID of the result in the Scopus database

—