BUT-FIT at SemEval-2020 Task 4: Multilingual commonsense
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26230%2F20%3APU138894" target="_blank" >RIV/00216305:26230/20:PU138894 - isvavai.cz</a>
Result on the web
<a href="https://www.aclweb.org/anthology/2020.semeval-1.46/" target="_blank" >https://www.aclweb.org/anthology/2020.semeval-1.46/</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
BUT-FIT at SemEval-2020 Task 4: Multilingual commonsense
Original language description
We participated in all three subtasks. In subtasks A and B, our submissions are based on pretrained language representation models (namely ALBERT) and data augmentation. We experimented with solving the task for another language, Czech, by means of multilingual models and machine translated dataset, or translated model inputs. We show that with a strong machine translation system, our system can be used in another language with a small accuracy loss. In subtask C, our submission, which is based on pretrained sequence-to-sequence model (BART), ranked 1st in BLEU score ranking, however, we show that the correlation between BLEU and human evaluation, in which our submission ended up 4th, is low. We analyse the metrics used in the evaluation and we propose an additional score based on model from subtask B, which correlates well with our manual ranking, as well as reranking method based on the same principle. We performed an error and dataset analysis for all subtasks and we present our findings.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/LTC18054" target="_blank" >LTC18054: Distant Reading for European Literary History</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2020
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the Fourteenth Workshop on Semantic Evaluation
ISBN
978-1-952148-31-6
ISSN
—
e-ISSN
—
Number of pages
16
Pages from-to
374-390
Publisher name
Association for Computational Linguistics
Place of publication
Barcelona
Event location
Barcelona (online)
Event date
Dec 8, 2020
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—