Words? Burstiness in Language Models

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14330%2F11%3A00067069" target="_blank" >RIV/00216224:14330/11:00067069 - isvavai.cz</a>
Result on the web
—
DOI - Digital Object Identifier
—

Result language
angličtina
Original language name
Words? Burstiness in Language Models
Original language description
Good estimation of the probability of a single word is a crucial part of language modelling. It is based on raw frequency of the word in a training corpus. Such computation is a good estimation for functional words and most very frequent words, but it isa poor estimation for most content words because of words' tendency to occur in clusters. This paper provides an analysis of words' burstiness and propose a new unigram language model which handles bursty words much better. The evaluation of the model on two data sets shows consistently lower perplexity and cross-entropy in the new model.
Czech name
—
Czech description
—

Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year
2011
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection
Proceedings of Recent Advances in Slavonic Natural Language Processing, RASLAN 2011
ISBN
9788026300779
ISSN
—
e-ISSN
—
Number of pages
7
Pages from-to
131-137
Publisher name
Tribun EU
Place of publication
Brno
Event location
Karlova Studánka
Event date
Dec 2, 2011
Type of event by nationality
EUR - Evropská akce
UT code for WoS article
—

Similar results(10)