Human–machine interaction in building an English reference dataset for natural language processing tasks

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3A4PSJHE7L" target="_blank" >RIV/00216208:11320/26:4PSJHE7L - isvavai.cz</a>

Result on the web

<a href="http://dx.doi.org/10.1007/s10579-025-09835-2" target="_blank" >http://dx.doi.org/10.1007/s10579-025-09835-2</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s10579-025-09835-2" target="_blank" >10.1007/s10579-025-09835-2</a>

Result language

angličtina

Original language name

Human–machine interaction in building an English reference dataset for natural language processing tasks

Original language description

Rich in information and annotated instances, a reference annotated dataset is essential for the training and evaluation of Natural Language Processing (NLP) tools. However, the creation of such linguistic resources is a tedious and time-consuming task involving lexical, syntactic, and semantic annotations, typically at the sentence level. Assuming we could speed up the human annotation process, we employed pre-trained models (spaCy, AllenNLP, EWISER) to automatically annotate a dataset of 664 sentences (6853 tokens, including 1598 predicates) taken from grammar books. A multi-layered annotation task encompassed Lemmatization (LEM), Part-of-Speech Tagging (UPOS, XPOS), Named Entity Recognition (NER), Dependency Parsing (DEP, HEAD), Coreference Resolution (COREF), Semantic Role Labelling (SRL), Predicate Sense Disambiguation (PSD) and Word Sense Disambiguation (WSD). Three annotators post-edited the noisy automatic annotations, and their average Inter-Annotator Agreement (IAA) for all annotation tasks at the token level was 0.91 and at the sentence level 0.74. Evaluation metrics including Accuracy, Precision, Recall, and F1 revealed disparities between machine and human annotations, along with correlations between machine annotations at both token and sentence levels. Manual error analysis identified instances where NLP tools failed to generate accurate annotations. A comparison of time spent per layer revealed that refining a pre-annotated subset of sentences required significantly less time than annotating them manually from scratch. This process resulted in an English reference dataset, tailored for the development of a hypergraph-based knowledge extraction model, known as the Natural Language 2 Semantic Hyper-graph Dataset (NL2SH) 1.0), which is accessible through CLARIN. © The Author(s), under exclusive licence to Springer Nature B.V. 2025.

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Language Resources and Evaluation

ISSN

1574-020X

e-ISSN

—

Volume of the periodical

59

Issue of the periodical within the volume

3

Country of publishing house

US - UNITED STATES

Number of pages

29

Pages from-to

2781-2809

UT code for WoS article

001484618500001

EID of the result in the Scopus database

2-s2.0-105004708583