Evaluating Attribution Methods for Explainable NLP with Transformers

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F22%3A43965893" target="_blank" >RIV/49777513:23520/22:43965893 - isvavai.cz</a>

Výsledek na webu

<a href="https://link.springer.com/chapter/10.1007/978-3-031-16270-1_1" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-031-16270-1_1</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-031-16270-1_1" target="_blank" >10.1007/978-3-031-16270-1_1</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Evaluating Attribution Methods for Explainable NLP with Transformers

Popis výsledku v původním jazyce

This paper describes the experimental evaluation of several attribution methods on two NLP tasks: Sentiment analysis and multi-label document classification. Our motivation is to find the best method to use with Transformers to interpret model decisions. For this purpose, we introduce two new evaluation datasets. The first one is derived from Stanford Sentiment Treebank, where the sentiment of individual words is annotated along with the sentiment of the whole sentence. The second dataset comes from Czech Text Document Corpus, where we added keyword information assigned to each category. The keywords were manually assigned to each document and automatically propagated to categories via PMI. We evaluate each attribution method on several models of different sizes. The evaluation results are reasonably consistent across all models and both datasets. It indicates that both datasets with proposed evaluation metrics are suitable for interpretability evaluation. We show how the attribution methods behave concerning model size and task. We also consider practical applications -- we show that while some methods perform well, they can be replaced with slightly worse-performing methods requiring significantly less time to compute.

Název v anglickém jazyce

Evaluating Attribution Methods for Explainable NLP with Transformers

Popis výsledku anglicky

This paper describes the experimental evaluation of several attribution methods on two NLP tasks: Sentiment analysis and multi-label document classification. Our motivation is to find the best method to use with Transformers to interpret model decisions. For this purpose, we introduce two new evaluation datasets. The first one is derived from Stanford Sentiment Treebank, where the sentiment of individual words is annotated along with the sentiment of the whole sentence. The second dataset comes from Czech Text Document Corpus, where we added keyword information assigned to each category. The keywords were manually assigned to each document and automatically propagated to categories via PMI. We evaluate each attribution method on several models of different sizes. The evaluation results are reasonably consistent across all models and both datasets. It indicates that both datasets with proposed evaluation metrics are suitable for interpretability evaluation. We show how the attribution methods behave concerning model size and task. We also consider practical applications -- we show that while some methods perform well, they can be replaced with slightly worse-performing methods requiring significantly less time to compute.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/TL03000152" target="_blank" >TL03000152: Umělá inteligence, média a právo</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2022

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

25th International Conference, TSD 2022, Brno, Czech Republic, September 6–9, 2022, Proceedings

ISBN

978-3-031-16269-5

ISSN

0302-9743

e-ISSN

1611-3349

Počet stran výsledku

12

Strana od-do

1-12

Název nakladatele

Springer

Místo vydání

Cham

Místo konání akce

Brno

Datum konání akce

6. 9. 2021

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

000866222300001