Comparison of text preprocessing methods
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3APHNVYRMF" target="_blank" >RIV/00216208:11320/23:PHNVYRMF - isvavai.cz</a>
Alternative codes found
RIV/00216208:11320/22:HXXS9Z4V
Result on the web
<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85161324782&doi=10.1017%2fS1351324922000213&partnerID=40&md5=3a4598d2c03a8beaa6fa1607525887e2" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85161324782&doi=10.1017%2fS1351324922000213&partnerID=40&md5=3a4598d2c03a8beaa6fa1607525887e2</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1017/s1351324922000213" target="_blank" >10.1017/s1351324922000213</a>
Alternative languages
Result language
angličtina
Original language name
Comparison of text preprocessing methods
Original language description
"Text preprocessing is not only an essential step to prepare the corpus for modeling but also a key area that directly affects the natural language processing (NLP) application results. For instance, precise tokenization increases the accuracy of part-of-speech (POS) tagging, and retaining multiword expressions improves reasoning and machine translation. The text corpus needs to be appropriately preprocessed before it is ready to serve as the input to computer models. The preprocessing requirements depend on both the nature of the corpus and the NLP application itself, that is, what researchers would like to achieve from analyzing the data. Conventional text preprocessing practices generally suffice, but there exist situations where the text preprocessing needs to be customized for better analysis results. Hence, we discuss the pros and cons of several common text preprocessing methods: removing formatting, tokenization, text normalization, handling punctuation, removing stopwords, stemming and lemmatization, n-gramming, and identifying multiword expressions. Then, we provide examples of text datasets which require special preprocessing and how previous researchers handled the challenge. We expect this article to be a starting guideline on how to select and fine-tune text preprocessing methods. © The Author(s), 2022. Published by Cambridge University Press."
Czech name
—
Czech description
—
Classification
Type
J<sub>SC</sub> - Article in a specialist periodical, which is included in the SCOPUS database
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2023
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Name of the periodical
"Natural Language Engineering"
ISSN
1351-3249
e-ISSN
—
Volume of the periodical
29
Issue of the periodical within the volume
3
Country of publishing house
US - UNITED STATES
Number of pages
45
Pages from-to
509-553
UT code for WoS article
—
EID of the result in the Scopus database
2-s2.0-85161324782