Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F20%3A10424441" target="_blank" >RIV/00216208:11320/20:10424441 - isvavai.cz</a>
Result on the web
<a href="https://www.aclweb.org/anthology/2020.inlg-1.19/" target="_blank" >https://www.aclweb.org/anthology/2020.inlg-1.19/</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
Original language description
A major challenge in evaluating data-to-text (D2T) generation is measuring the semantic accuracy of the generated text, i.e. its faithfulness to the input data. We propose a new metric for evaluating the semantic accuracy of D2T generation based on a neural model pretrained for natural language inference (NLI). We use the NLI model to check textual entailment between the input data and the output text in both directions, allowing us to reveal omissions or hallucinations. Input data are converted to text for NLI using trivial templates. Our experiments on two recent D2T datasets show that our metric can achieve high accuracy in identifying erroneous system outputs.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
S - Specificky vyzkum na vysokych skolach<br>I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2020
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 13th International Conference on Natural Language Generation (INLG 2020)
ISBN
978-1-952148-54-5
ISSN
—
e-ISSN
—
Number of pages
7
Pages from-to
131-137
Publisher name
Association for Computational Linguistics
Place of publication
Stroudsburgh, PA, USA
Event location
Online
Event date
Dec 15, 2020
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—