Evaluating Attribution Methods for Explainable NLP with Transformers
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F22%3A43965893" target="_blank" >RIV/49777513:23520/22:43965893 - isvavai.cz</a>
Result on the web
<a href="https://link.springer.com/chapter/10.1007/978-3-031-16270-1_1" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-031-16270-1_1</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/978-3-031-16270-1_1" target="_blank" >10.1007/978-3-031-16270-1_1</a>
Alternative languages
Result language
angličtina
Original language name
Evaluating Attribution Methods for Explainable NLP with Transformers
Original language description
This paper describes the experimental evaluation of several attribution methods on two NLP tasks: Sentiment analysis and multi-label document classification. Our motivation is to find the best method to use with Transformers to interpret model decisions. For this purpose, we introduce two new evaluation datasets. The first one is derived from Stanford Sentiment Treebank, where the sentiment of individual words is annotated along with the sentiment of the whole sentence. The second dataset comes from Czech Text Document Corpus, where we added keyword information assigned to each category. The keywords were manually assigned to each document and automatically propagated to categories via PMI. We evaluate each attribution method on several models of different sizes. The evaluation results are reasonably consistent across all models and both datasets. It indicates that both datasets with proposed evaluation metrics are suitable for interpretability evaluation. We show how the attribution methods behave concerning model size and task. We also consider practical applications -- we show that while some methods perform well, they can be replaced with slightly worse-performing methods requiring significantly less time to compute.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/TL03000152" target="_blank" >TL03000152: Artificial Intelligence, Media and Law</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2022
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
25th International Conference, TSD 2022, Brno, Czech Republic, September 6–9, 2022, Proceedings
ISBN
978-3-031-16269-5
ISSN
0302-9743
e-ISSN
1611-3349
Number of pages
12
Pages from-to
1-12
Publisher name
Springer
Place of publication
Cham
Event location
Brno
Event date
Sep 6, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
000866222300001