Lemmatization of low-resource languages in diachronic linguistics: Problems and solutions

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AFJXCVJM6" target="_blank" >RIV/00216208:11320/26:FJXCVJM6 - isvavai.cz</a>

Result on the web

<a href="https://lib.herzen.spb.ru/media/magazines/contents/1/217/25_drozashchikh_217_302_311.pdf" target="_blank" >https://lib.herzen.spb.ru/media/magazines/contents/1/217/25_drozashchikh_217_302_311.pdf</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.33910/1992-6464-2025-217-302-311" target="_blank" >10.33910/1992-6464-2025-217-302-311</a>

Result language

ruština

Original language name

Lemmatization of low-resource languages in diachronic linguistics: Problems and solutions

Original language description

Introduction. The article addresses the problem of lemmatizing low-resource historical languages within applied diachronic linguistics. Standard approaches widely used for modern texts — rule-based models and neural architectures — are inapplicable for ancient languages because of their morpho- logical complexity and the scarcity of corpus data. As a result, dictionary-based lemmatization remains the most effective strategy. For Old English, the widely available Classical Language Toolkit (CLTK) offers only limited functionality, since its lemma dictionary provides insufficient coverage. The aim of this study is to compile an expanded lemma dictionary of Old English using data from the crowd- sourced lexicographic resource Wiktionary, thereby enabling more accurate automatic lemmatization. Materials and Methods. The study draws on the annotated treebank corpus of Old English texts (9th–11th centuries) and the datasets from the open lexicographic resources (the CLTK lemma list and the Old English segment of Wiktionary). The methodological framework combines methods of corpus and computational linguistics, and electronic lexicography. The research consists of three stages: compilation of an Old English lemma dictionary, lemmatization of Old English texts, and evaluation of the dictionary’s precision and recall. Results. The analysis of the lemmatizers for low-resource languages demonstrates that the dictiona- ry-based approach proves to be more effective. Traditional lexicographic resources such as Bosworth– Toller remain valuable, but their lack of machine-readable formats limits direct application. As an al- ternative we propose Wiktionary — a free, crowdsourced lexicographic resource characterized by both broad lexical coverage and detailed descriptions of lemmas. Within the framework of this study, we developed a compiled lemma dictionary by integrating CLTK lemma list and Wiktionary datasets. The resulting lemma dictionary (11,451 unique lemmas, 80,778 wordforms) demonstrates a high degree of precision and recall, confirming its applicability for the lemmatization of Old English texts. Conclusion. The study thus provides a new digital resource — an Old English lemma dictionary accompanied by Python code for fully automatic lemmatization. Together, these tools represent an ef- fective solution to the lemmatization of a low-resource historical language and contribute to the develop- ment of computational diachronic linguistics.

Czech name

—

Czech description

—

Type

J_ost - Miscellaneous article in a specialist periodical

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Izvestia: Herzen University Journal of Humanities & Sciences

ISSN

19926464

e-ISSN

—

Volume of the periodical

2025

Issue of the periodical within the volume

217

Country of publishing house

RU - RUSSIAN FEDERATION

Number of pages

10

Pages from-to

302-311

UT code for WoS article

—

EID of the result in the Scopus database

—