Lexical Normalization Using Generative Transformer Model (LN-GTM)

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3AS6IHBN47" target="_blank" >RIV/00216208:11320/23:S6IHBN47 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85176453443&doi=10.1007%2fs44196-023-00366-8&partnerID=40&md5=d14a281d74227cd320a3f33a9720193d" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85176453443&doi=10.1007%2fs44196-023-00366-8&partnerID=40&md5=d14a281d74227cd320a3f33a9720193d</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s44196-023-00366-8" target="_blank" >10.1007/s44196-023-00366-8</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Lexical Normalization Using Generative Transformer Model (LN-GTM)

Popis výsledku v původním jazyce

"Lexical Normalization (LN) aims to normalize a nonstandard text to a standard text. This problem is of extreme importance in natural language processing (NLP) when applying existing trained models to user-generated text on social media. Users of social media tend to use non-standard language. They heavily use abbreviations, phonetic substitutions, and colloquial language. Nevertheless, most existing NLP-based systems are often designed with the standard language in mind. However, they suffer from significant performance drops due to the many out-of-vocabulary words found in social media text. In this paper, we present a new (LN) technique by utilizing a transformer-based sequence-to-sequence (Seq2Seq) to build a multilingual characters-to-words machine translation model. Unlike the majority of current methods, the proposed model is capable of recognizing and generating previously unseen words. Also, it greatly reduces the difficulties involved in tokenizing and preprocessing the nonstandard text input and the standard text output. The proposed model outperforms the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021 on both intrinsic and extrinsic evaluations. © 2023, The Author(s)."

Název v anglickém jazyce

Lexical Normalization Using Generative Transformer Model (LN-GTM)

Popis výsledku anglicky

"Lexical Normalization (LN) aims to normalize a nonstandard text to a standard text. This problem is of extreme importance in natural language processing (NLP) when applying existing trained models to user-generated text on social media. Users of social media tend to use non-standard language. They heavily use abbreviations, phonetic substitutions, and colloquial language. Nevertheless, most existing NLP-based systems are often designed with the standard language in mind. However, they suffer from significant performance drops due to the many out-of-vocabulary words found in social media text. In this paper, we present a new (LN) technique by utilizing a transformer-based sequence-to-sequence (Seq2Seq) to build a multilingual characters-to-words machine translation model. Unlike the majority of current methods, the proposed model is capable of recognizing and generating previously unseen words. Also, it greatly reduces the difficulties involved in tokenizing and preprocessing the nonstandard text input and the standard text output. The proposed model outperforms the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021 on both intrinsic and extrinsic evaluations. © 2023, The Author(s)."

Druh

J_SC - Článek v periodiku v databázi SCOPUS

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

"International Journal of Computational Intelligence Systems"

ISSN

1875-6891

e-ISSN

—

Svazek periodika

16

Číslo periodika v rámci svazku

1

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

11

Strana od-do

1-11

Kód UT WoS článku

—

EID výsledku v databázi Scopus

2-s2.0-85176453443