Czert – Czech BERT-like Model for Language Representation
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F21%3A43962619" target="_blank" >RIV/49777513:23520/21:43962619 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2021.ranlp-1.149/" target="_blank" >https://aclanthology.org/2021.ranlp-1.149/</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.26615/978-954-452-072-4_149" target="_blank" >10.26615/978-954-452-072-4_149</a>
Alternative languages
Result language
angličtina
Original language name
Czert – Czech BERT-like Model for Language Representation
Original language description
This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures. We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data. We outperform the multilingual models on 9 out of 11 datasets. In addition, we establish the new state-of-the-art results on nine datasets. At the end, we discuss properties of monolingual and multilingual models based upon our results. We publish all the pre-trained and fine-tuned models freely for the research community.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/EF17_048%2F0007267" target="_blank" >EF17_048/0007267: Research and Development of Intelligent Components of Advanced Technologies for the Pilsen Metropolitan Area (InteCom)</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach<br>I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Deep Learning for Natural Language Processing Methods and Applications
ISBN
978-954-452-072-4
ISSN
1313-8502
e-ISSN
—
Number of pages
13
Pages from-to
1326-1338
Publisher name
INCOMA, Ltd.
Place of publication
Shoumen
Event location
Shoumen, Bulgaria
Event date
Sep 1, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—