Efficient Text Classification with Echo State Networks

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F67985807%3A_____%2F21%3A00542211" target="_blank" >RIV/67985807:_____/21:00542211 - isvavai.cz</a>

Výsledek na webu

<a href="http://dx.doi.org/10.1109/IJCNN52387.2021.9533958" target="_blank" >http://dx.doi.org/10.1109/IJCNN52387.2021.9533958</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/IJCNN52387.2021.9533958" target="_blank" >10.1109/IJCNN52387.2021.9533958</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Efficient Text Classification with Echo State Networks

Popis výsledku v původním jazyce

We consider echo state networks (ESNs) for text classification. More specifically, we investigate the learning capabilities of ESNs with pre-trained word embedding as input features, trained on the IMDb and TREC sentiment and question classification datasets, respectively. First, we introduce a customized training paradigm for the processing of multiple input time series (the inputs texts) associated with categorical targets (their corresponding classes). For sentiment tasks, we use an additional frozen attention mechanism which is based on an external lexicon, and hence requires only negligible computational cost. Within this paradigm, ESNs can be trained in tens of seconds on a GPU. We show that ESNs significantly outperform their Ridge regression baselines provided with the same embedded features. ESNs also compete with classical Bi-LSTM networks while keeping a training time of up to 23 times faster. These results show that ESNs can be considered as robust, efficient and fast candidates for text classification tasks. Overall, this study falls within the context of light and fast-to-train models for NLP.

Název v anglickém jazyce

Efficient Text Classification with Echo State Networks

Popis výsledku anglicky

We consider echo state networks (ESNs) for text classification. More specifically, we investigate the learning capabilities of ESNs with pre-trained word embedding as input features, trained on the IMDb and TREC sentiment and question classification datasets, respectively. First, we introduce a customized training paradigm for the processing of multiple input time series (the inputs texts) associated with categorical targets (their corresponding classes). For sentiment tasks, we use an additional frozen attention mechanism which is based on an external lexicon, and hence requires only negligible computational cost. Within this paradigm, ESNs can be trained in tens of seconds on a GPU. We show that ESNs significantly outperform their Ridge regression baselines provided with the same embedded features. ESNs also compete with classical Bi-LSTM networks while keeping a training time of up to 23 times faster. These results show that ESNs can be considered as robust, efficient and fast candidates for text classification tasks. Overall, this study falls within the context of light and fast-to-train models for NLP.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

<a href="/cs/project/GA19-05704S" target="_blank" >GA19-05704S: FoNeCo: Analytické základy neurovýpočtů</a><br>

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

IJCNN 2021. The International Joint Conference on Neural Networks Proceedings

ISBN

978-0-7381-3366-9

ISSN

—

e-ISSN

—

Počet stran výsledku

8

Strana od-do

1-8

Název nakladatele

IEEE

Místo vydání

Piscataway

Místo konání akce

Virtual

Datum konání akce

18. 7. 2021

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

000722581705038