BanLemma: A Word Formation Dependent Rule and Dictionary Based Bangla Lemmatizer
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A6VSVYWFA" target="_blank" >RIV/00216208:11320/23:6VSVYWFA - isvavai.cz</a>
Result on the web
<a href="http://arxiv.org/abs/2311.03078" target="_blank" >http://arxiv.org/abs/2311.03078</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.48550/arXiv.2311.03078" target="_blank" >10.48550/arXiv.2311.03078</a>
Alternative languages
Result language
angličtina
Original language name
BanLemma: A Word Formation Dependent Rule and Dictionary Based Bangla Lemmatizer
Original language description
"Lemmatization holds significance in both natural language processing (NLP) and linguistics, as it effectively decreases data density and aids in comprehending contextual meaning. However, due to the highly inflected nature and morphological richness, lemmatization in Bangla text poses a complex challenge. In this study, we propose linguistic rules for lemmatization and utilize a dictionary along with the rules to design a lemmatizer specifically for Bangla. Our system aims to lemmatize words based on their parts of speech class within a given sentence. Unlike previous rule-based approaches, we analyzed the suffix marker occurrence according to the morpho-syntactic values and then utilized sequences of suffix markers instead of entire suffixes. To develop our rules, we analyze a large corpus of Bangla text from various domains, sources, and time periods to observe the word formation of inflected words. The lemmatizer achieves an accuracy of 96.36% when tested against a manually annotated test dataset by trained linguists and demonstrates competitive performance on three previously published Bangla lemmatization datasets. We are making the code and datasets publicly available at https://github.com/eblict-gigatech/BanLemma in order to contribute to the further advancement of Bangla NLP."
Czech name
—
Czech description
—
Classification
Type
O - Miscellaneous
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
—
Others
Publication year
2023
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů