An Untold Story of Preprocessing Task Evaluation: An Alignment-based Joint Evaluation Approach

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AY3C5NGAV" target="_blank" >RIV/00216208:11320/25:Y3C5NGAV - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195890411&partnerID=40&md5=02c652c2e07770a28862bcfc2515c983" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195890411&partnerID=40&md5=02c652c2e07770a28862bcfc2515c983</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

An Untold Story of Preprocessing Task Evaluation: An Alignment-based Joint Evaluation Approach

Original language description

A preprocessing task such as tokenization and sentence boundary detection (SBD) has commonly been considered as NLP challenges that have already been solved. This perception is due to their generally good performance and the presence of pre-tokenized data. However, it's important to note that the low error rates of current methods are mainly specific to certain tasks, and rule-based tokenization can be difficult to use across different systems. Despite being subtle, these limitations are significant in the context of the NLP pipeline. In this paper, we introduce a novel evaluation algorithm for the preprocessing task, including both tokenization and SBD results. This algorithm aims to enhance the reliability of evaluations by reevaluating the counts of true positive cases for F1 measures in both preprocessing tasks jointly. It achieves this through an alignment-based approach inspired by sentence and word alignments used in machine translation. Our evaluation algorithm not only allows for precise counting of true positive tokens and sentence boundaries but also combines these two evaluation tasks into a single organized pipeline. To illustrate and clarify the intricacies of this calculation and integration, we provide detailed pseudo-code configurations for implementation. Additionally, we offer empirical evidence demonstrating how sentence and word alignment can improve evaluation reliability and present case studies to further support our approach. © 2024 ELRA Language Resource Association: CC BY-NC 4.0.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Jt. Int. Conf. Comput. Linguist., Lang. Resour. Eval., LREC-COLING - Main Conf. Proc.

ISBN

978-249381410-4

ISSN

—

e-ISSN

—

Number of pages

12

Pages from-to

1327-1338

Publisher name

European Language Resources Association (ELRA)

Place of publication

—

Event location

Torino, Italia

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—