Comparison of Czech Transformers on Text Classification Tasks

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F21%3A43962816" target="_blank" >RIV/49777513:23520/21:43962816 - isvavai.cz</a>
Result on the web
<a href="https://link.springer.com/chapter/10.1007/978-3-030-89579-2_3" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-030-89579-2_3</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/978-3-030-89579-2_3" target="_blank" >10.1007/978-3-030-89579-2_3</a>

Result language
angličtina
Original language name
Comparison of Czech Transformers on Text Classification Tasks
Original language description
In this paper, we present our progress in pre-training monolingual Transformers for Czech and contribute to the research community by releasing our models for public. The need for such models emerged from our effort to employ Transformers in our language-specific tasks, but we found the performance of the published multilingual models to be very limited. Since the multilingual models are usually pre-trained from 100+ languages, most of low-resourced languages (including Czech) are under-represented in these models. At the same time, there is a huge amount of monolingual training data available in web archives like Common Crawl. We have pre-trained and publicly released two monolingual Czech Transformers and compared them with relevant public models, trained (at least partially) for Czech. The paper presents the Transformers pre-training procedure as well as a comparison of pre-trained models on text classification task from various domains.
Czech name
—
Czech description
—

Project
<a href="/en/project/DG18P02OVV016" target="_blank" >DG18P02OVV016: Development of the centralized interface for the web content and social networks data mining.</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection
Statistical Language and Speech Processing, SLSP 2021
ISBN
978-3-030-89578-5
ISSN
0302-9743
e-ISSN
1611-3349
Number of pages
11
Pages from-to
27-37
Publisher name
Springer
Place of publication
Cham
Event location
Cardiff, UK
Event date
Nov 23, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—

Similar results(10)