The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F21%3A10440535" target="_blank" >RIV/00216208:11320/21:10440535 - isvavai.cz</a>

Výsledek na webu

<a href="https://aclanthology.org/2021.gem-1.10/" target="_blank" >https://aclanthology.org/2021.gem-1.10/</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2021.gem-1.10" target="_blank" >10.18653/v1/2021.gem-1.10</a>

Jazyk výsledku

angličtina

Název v původním jazyce

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Popis výsledku v původním jazyce

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

Název v anglickém jazyce

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Popis výsledku anglicky

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2021

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

ISBN

978-1-954085-67-1

ISSN

—

e-ISSN

—

Počet stran výsledku

25

Strana od-do

96-120

Název nakladatele

Association for Computational Linguistics

Místo vydání

Stroudsburg, PA, USA

Místo konání akce

Online

Datum konání akce

1. 8. 2021

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—