Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AWLPZG35Q" target="_blank" >RIV/00216208:11320/25:WLPZG35Q - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195113264&partnerID=40&md5=8717e05c3e3fb7a8931831ad346e5c53" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195113264&partnerID=40&md5=8717e05c3e3fb7a8931831ad346e5c53</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks

Original language description

The Parallel Meaning Bank (PMB) serves as a corpus for semantic processing with a focus on semantic parsing and text generation. Currently, we witness an excellent performance of neural parsers and generators on the PMB. This might suggest that such semantic processing tasks have by and large been solved. We argue that this is not the case and that performance scores from the past on the PMB are inflated by non-optimal data splits and test sets that are too easy. In response, we introduce several changes. First, instead of the prior random split, we propose a more systematic splitting approach to improve the reliability of the standard test data. Second, except for the standard test set, we also propose two challenge sets: one with longer texts including discourse structure, and one that addresses compositional generalization. We evaluate five neural models for semantic parsing and meaning-to-text generation. Our results show that model performance declines (in some cases dramatically) on the challenge sets, revealing the limitations of neural models when confronting such challenges. © 2024 ELRA Language Resource Association.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Int. Workshop Des. Mean. Represent., DMR LREC-COLING - Workshop Proc.

ISBN

978-249381439-5

ISSN

—

e-ISSN

—

Number of pages

14

Pages from-to

162-175

Publisher name

European Language Resources Association (ELRA)

Place of publication

—

Event location

Torino, Italia

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—