General framework for mining, processing and storing large amounts of electronic texts for language modeling purposes

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F14%3A43919601" target="_blank" >RIV/49777513:23520/14:43919601 - isvavai.cz</a>

Výsledek na webu

<a href="http://dx.doi.org/10.1007/s10579-013-9246-z" target="_blank" >http://dx.doi.org/10.1007/s10579-013-9246-z</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/s10579-013-9246-z" target="_blank" >10.1007/s10579-013-9246-z</a>

Jazyk výsledku

angličtina

Název v původním jazyce

General framework for mining, processing and storing large amounts of electronic texts for language modeling purposes

Popis výsledku v původním jazyce

The paper describes a general framework for mining large amounts of text data from a defined set of Web pages. The acquired data are meant to constitute a corpus for training robust and reliable language models and thus the framework needs to also incorporate algorithms for appropriate text processing and duplicity detection in order to secure quality and consistency of the data. As we expect the resulting corpus to be very large, we have also implemented topic detection algorithms that allow us to automatically select subcorpora for domain-specific language models. The description of the framework architecture and the implemented algorithms is complemented with a detailed evaluation section. It analyses the basic properties of the gathered Czech corpus containing more than one billion text tokens collected using the described framework, shows the results of the topic detection methods and finally also describes the design and outcomes of the automatic speech recognition experiments wi

Název v anglickém jazyce

General framework for mining, processing and storing large amounts of electronic texts for language modeling purposes

Popis výsledku anglicky

The paper describes a general framework for mining large amounts of text data from a defined set of Web pages. The acquired data are meant to constitute a corpus for training robust and reliable language models and thus the framework needs to also incorporate algorithms for appropriate text processing and duplicity detection in order to secure quality and consistency of the data. As we expect the resulting corpus to be very large, we have also implemented topic detection algorithms that allow us to automatically select subcorpora for domain-specific language models. The description of the framework architecture and the implemented algorithms is complemented with a detailed evaluation section. It analyses the basic properties of the gathered Czech corpus containing more than one billion text tokens collected using the described framework, shows the results of the topic detection methods and finally also describes the design and outcomes of the automatic speech recognition experiments wi

Druh

J_x - Nezařazeno - Článek v odborném periodiku (Jimp, Jsc a Jost)

CEP obor

IN - Informatika

OECD FORD obor

—

Projekt

<a href="/cs/project/GBP103%2F12%2FG084" target="_blank" >GBP103/12/G084: Centrum pro multi-modální interpretaci dat velkého rozsahu</a><br>

Návaznosti

S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2014

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Language Resources and Evaluation

ISSN

1574-020X

e-ISSN

—

Svazek periodika

48

Číslo periodika v rámci svazku

2

Stát vydavatele periodika

NL - Nizozemsko

Počet stran výsledku

22

Strana od-do

227-248

Kód UT WoS článku

000335779200003

EID výsledku v databázi Scopus

—