Language Identification as part of the Text Corpus Creation Pipeline at the Language Bank of Finland

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F22%3AIBZCDHSZ" target="_blank" >RIV/00216208:11320/22:IBZCDHSZ - isvavai.cz</a>

Result on the web

<a href="https://helda.helsinki.fi/handle/10138/350012" target="_blank" >https://helda.helsinki.fi/handle/10138/350012</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Language Identification as part of the Text Corpus Creation Pipeline at the Language Bank of Finland

Original language description

The Language Bank of Finland hosts text corpora originating from Finland. Two of the most used ones are the Newspaper and Periodical Corpus of the National Library of Finland and the Suomi24 Corpus. The Language Bank has received considerable additions to both corpora and is currently creating new versions of the corpora. We are debuting language identification as part of the corpus creation pipeline. As a language identifier, we are using our recently published HeLI-OTS software. This paper investigates the results and the quality of language identification. We created a new dataset for evaluating the efficacy of language identification by extracting random samples from both corpora and manually annotating their language. We were especially interested in seeing how the relatively low OCR quality of the oldest part of the KLK-fi collection will affect language identification when using an off-the-shelf program like HeLI-OTS. The oldest part of the KLK-fi collection and many parts of the Suomi24 corpus contain Finnish written dialectally or otherwise differing from the standard written Finnish. HeLI-OTS software includes several separate language models for dialectal Finnish, and in this article, we evaluate their usefulness using the new data set. For both corpora, the overall micro F1 scores increase when using the additional dialectal models. Additionally, we take a detailed look at some language identification errors and discuss possible solutions.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2022

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

The 6th Digital Humanities in the Nordic and Baltic Countries Conference (DHNB 2022)

ISBN

—

ISSN

1613-0073

e-ISSN

—

Number of pages

9

Pages from-to

251-259

Publisher name

CEUR

Place of publication

—

Event location

Uppsala, Sweden

Event date

Jan 1, 2022

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—