Part-of-speech Tagging for Extremely Low-resource Indian Languages

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3AU37HDDEJ" target="_blank" >RIV/00216208:11320/25:U37HDDEJ - isvavai.cz</a>

Result on the web

<a href="https://www.scopus.com/inward/record.uri?eid=2-s2.0-85205281827&partnerID=40&md5=e92d325d2aa44d3fc9c7c8081bf888f4" target="_blank" >https://www.scopus.com/inward/record.uri?eid=2-s2.0-85205281827&partnerID=40&md5=e92d325d2aa44d3fc9c7c8081bf888f4</a>

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Part-of-speech Tagging for Extremely Low-resource Indian Languages

Original language description

Modern natural language processing (NLP) systems thrive when given access to large datasets. However, a large fraction of the world's languages are not privy to such benefits due to sparse documentation and inadequate digital representation. This is especially true for Indian regional languages. As a first step towards expanding the reach of NLP technologies to extremely low-resource Indian languages, we present a new parallel part-of-speech (POS) evaluation dataset for Angika, Magahi, Bhojpuri and Hindi. Angika, Magahi, Bhojpuri, along with the more well-known Hindi, are all languages spoken in the Indian states of Bihar, Jharkhand and West Bengal. Ours is notably the first NLP resource, even for a shallow NLP task like POS-tagging, for Angika. We establish POS-tagging baselines using state-of-the-art multilingual pretrained language models (PLMs) finetuned on Hindi data, and show zero-shot evaluations on the other three languages. While all four languages use the same Devanagari script, pretrained tokenizers underperform in zero-shot on the three languages. We propose a simple look-back fix to address the tokenization challenge yielding F1-score improvements of up to 8% on Angika, and show how it comes very close to an oracle setting when the underlying Hindi word is known (and can be accurately tokenized). © 2024 Association for Computational Linguistics.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proc. Annu. Meet. Assoc. Comput Linguist.

ISBN

979-889176099-8

ISSN

0736-587X

e-ISSN

—

Number of pages

10

Pages from-to

14422-14431

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

—

Event location

Hybrid, Bangkok

Event date

Jan 1, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—