Lemmatization of low-resource languages in diachronic linguistics: Problems and solutions

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AFJXCVJM6" target="_blank" >RIV/00216208:11320/26:FJXCVJM6 - isvavai.cz</a>

Výsledek na webu

<a href="https://lib.herzen.spb.ru/media/magazines/contents/1/217/25_drozashchikh_217_302_311.pdf" target="_blank" >https://lib.herzen.spb.ru/media/magazines/contents/1/217/25_drozashchikh_217_302_311.pdf</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.33910/1992-6464-2025-217-302-311" target="_blank" >10.33910/1992-6464-2025-217-302-311</a>

Jazyk výsledku

ruština

Název v původním jazyce

Lemmatization of low-resource languages in diachronic linguistics: Problems and solutions

Popis výsledku v původním jazyce

Introduction. The article addresses the problem of lemmatizing low-resource historical languages within applied diachronic linguistics. Standard approaches widely used for modern texts — rule-based models and neural architectures — are inapplicable for ancient languages because of their morpho- logical complexity and the scarcity of corpus data. As a result, dictionary-based lemmatization remains the most effective strategy. For Old English, the widely available Classical Language Toolkit (CLTK) offers only limited functionality, since its lemma dictionary provides insufficient coverage. The aim of this study is to compile an expanded lemma dictionary of Old English using data from the crowd- sourced lexicographic resource Wiktionary, thereby enabling more accurate automatic lemmatization. Materials and Methods. The study draws on the annotated treebank corpus of Old English texts (9th–11th centuries) and the datasets from the open lexicographic resources (the CLTK lemma list and the Old English segment of Wiktionary). The methodological framework combines methods of corpus and computational linguistics, and electronic lexicography. The research consists of three stages: compilation of an Old English lemma dictionary, lemmatization of Old English texts, and evaluation of the dictionary’s precision and recall. Results. The analysis of the lemmatizers for low-resource languages demonstrates that the dictiona- ry-based approach proves to be more effective. Traditional lexicographic resources such as Bosworth– Toller remain valuable, but their lack of machine-readable formats limits direct application. As an al- ternative we propose Wiktionary — a free, crowdsourced lexicographic resource characterized by both broad lexical coverage and detailed descriptions of lemmas. Within the framework of this study, we developed a compiled lemma dictionary by integrating CLTK lemma list and Wiktionary datasets. The resulting lemma dictionary (11,451 unique lemmas, 80,778 wordforms) demonstrates a high degree of precision and recall, confirming its applicability for the lemmatization of Old English texts. Conclusion. The study thus provides a new digital resource — an Old English lemma dictionary accompanied by Python code for fully automatic lemmatization. Together, these tools represent an ef- fective solution to the lemmatization of a low-resource historical language and contribute to the develop- ment of computational diachronic linguistics.

Název v anglickém jazyce

Lemmatization of low-resource languages in diachronic linguistics: Problems and solutions

Popis výsledku anglicky

Introduction. The article addresses the problem of lemmatizing low-resource historical languages within applied diachronic linguistics. Standard approaches widely used for modern texts — rule-based models and neural architectures — are inapplicable for ancient languages because of their morpho- logical complexity and the scarcity of corpus data. As a result, dictionary-based lemmatization remains the most effective strategy. For Old English, the widely available Classical Language Toolkit (CLTK) offers only limited functionality, since its lemma dictionary provides insufficient coverage. The aim of this study is to compile an expanded lemma dictionary of Old English using data from the crowd- sourced lexicographic resource Wiktionary, thereby enabling more accurate automatic lemmatization. Materials and Methods. The study draws on the annotated treebank corpus of Old English texts (9th–11th centuries) and the datasets from the open lexicographic resources (the CLTK lemma list and the Old English segment of Wiktionary). The methodological framework combines methods of corpus and computational linguistics, and electronic lexicography. The research consists of three stages: compilation of an Old English lemma dictionary, lemmatization of Old English texts, and evaluation of the dictionary’s precision and recall. Results. The analysis of the lemmatizers for low-resource languages demonstrates that the dictiona- ry-based approach proves to be more effective. Traditional lexicographic resources such as Bosworth– Toller remain valuable, but their lack of machine-readable formats limits direct application. As an al- ternative we propose Wiktionary — a free, crowdsourced lexicographic resource characterized by both broad lexical coverage and detailed descriptions of lemmas. Within the framework of this study, we developed a compiled lemma dictionary by integrating CLTK lemma list and Wiktionary datasets. The resulting lemma dictionary (11,451 unique lemmas, 80,778 wordforms) demonstrates a high degree of precision and recall, confirming its applicability for the lemmatization of Old English texts. Conclusion. The study thus provides a new digital resource — an Old English lemma dictionary accompanied by Python code for fully automatic lemmatization. Together, these tools represent an ef- fective solution to the lemmatization of a low-resource historical language and contribute to the develop- ment of computational diachronic linguistics.

Druh

J_ost - Ostatní články v recenzovaných periodicích

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

—

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Izvestia: Herzen University Journal of Humanities & Sciences

ISSN

19926464

e-ISSN

—

Svazek periodika

2025

Číslo periodika v rámci svazku

217

Stát vydavatele periodika

RU - Ruská federace

Počet stran výsledku

10

Strana od-do

302-311

Kód UT WoS článku

—

EID výsledku v databázi Scopus

—