Where are we Still Split on Tokenization?

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A6FBSGVSC" target="_blank" >RIV/00216208:11320/25:6FBSGVSC - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85188745092&partnerID=40&md5=1ec486ce18b0cb9be7360d528093b48c" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85188745092&partnerID=40&md5=1ec486ce18b0cb9be7360d528093b48c</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Where are we Still Split on Tokenization?

Original language description

Many Natural Language Processing (NLP) tasks are labeled on the token level, for these tasks, the first step is to identify the tokens (tokenization). Because this step is often considered to be a solved problem, gold tokenization is commonly assumed. In this paper, we investigate if this task is solved with supervised tokenizers. To this end, we propose an effient multi-task model for tokenization that performs on-par with the state-of-the-art. We use this model to reflect on the status of performance on the tokenization task by evaluating on 122 languages in 20 different scripts. We show that tokenization performance is mainly dependent on the amount and consistency of annotated data as well as difficulty of the task in the writing systems. We conclude that besides inconsistencies in the data and exceptional cases the task can be considered solved for Latin languages for in-dataset settings (>99.5 F1). However, performance is 0.75 F1 point lower on average for datasets in other scripts and performance deteriorates in cross-dataset setups. © 2024 Association for Computational Linguistics.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

EACL - Conf. Eur. Chapter Assoc. Comput. Linguist., Find. EACL

ISBN

979-889176093-6

ISSN

—

e-ISSN

—

Number of pages

20

Pages from-to

118-137

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

—

Event location

St. Julian's

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—