The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F21%3A10440535" target="_blank" >RIV/00216208:11320/21:10440535 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2021.gem-1.10/" target="_blank" >https://aclanthology.org/2021.gem-1.10/</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.18653/v1/2021.gem-1.10" target="_blank" >10.18653/v1/2021.gem-1.10</a>
Alternative languages
Result language
angličtina
Original language name
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Original language description
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
ISBN
978-1-954085-67-1
ISSN
—
e-ISSN
—
Number of pages
25
Pages from-to
96-120
Publisher name
Association for Computational Linguistics
Place of publication
Stroudsburg, PA, USA
Event location
Online
Event date
Aug 1, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—