General framework for mining, processing and storing large amounts of electronic texts for language modeling purposes

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F14%3A43919601" target="_blank" >RIV/49777513:23520/14:43919601 - isvavai.cz</a>
Result on the web
<a href="http://dx.doi.org/10.1007/s10579-013-9246-z" target="_blank" >http://dx.doi.org/10.1007/s10579-013-9246-z</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/s10579-013-9246-z" target="_blank" >10.1007/s10579-013-9246-z</a>

Result language
angličtina
Original language name
General framework for mining, processing and storing large amounts of electronic texts for language modeling purposes
Original language description
The paper describes a general framework for mining large amounts of text data from a defined set of Web pages. The acquired data are meant to constitute a corpus for training robust and reliable language models and thus the framework needs to also incorporate algorithms for appropriate text processing and duplicity detection in order to secure quality and consistency of the data. As we expect the resulting corpus to be very large, we have also implemented topic detection algorithms that allow us to automatically select subcorpora for domain-specific language models. The description of the framework architecture and the implemented algorithms is complemented with a detailed evaluation section. It analyses the basic properties of the gathered Czech corpus containing more than one billion text tokens collected using the described framework, shows the results of the topic detection methods and finally also describes the design and outcomes of the automatic speech recognition experiments wi
Czech name
—
Czech description
—

Type
J<sub>x</sub> - Unclassified - Peer-reviewed scientific article (Jimp, Jsc and Jost)
CEP classification
IN - Informatics
OECD FORD branch
—

Project
<a href="/en/project/GBP103%2F12%2FG084" target="_blank" >GBP103/12/G084: Center for Large Scale Multi-modal Data Interpretation</a><br>
Continuities
S - Specificky vyzkum na vysokych skolach

Publication year
2014
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Similar results(10)