Web Text Data Mining for Building Large Scale Language Modelling Corpus

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F11%3A43898187" target="_blank" >RIV/49777513:23520/11:43898187 - isvavai.cz</a>

Result on the web

<a href="http://dx.doi.org/10.1007/978-3-642-23538-2_45" target="_blank" >http://dx.doi.org/10.1007/978-3-642-23538-2_45</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-642-23538-2_45" target="_blank" >10.1007/978-3-642-23538-2_45</a>

Result language

angličtina

Original language name

Web Text Data Mining for Building Large Scale Language Modelling Corpus

Original language description

The paper describes a system for collecting a large text corpus from Internet news servers. The architecture and text preprocessing algorithms are described. We also describe the used duplicity detection algorithm. The resulting corpus contains more than1 billion tokens in more than 3 millions articles with assigned topics and duplicates identified. Corpus statistics like consistency and perplexity are presented.

Czech name

—

Czech description

—

Type

J_x - Unclassified - Peer-reviewed scientific article (Jimp, Jsc and Jost)

CEP classification

JD - Use of computers, robotics and its application

OECD FORD branch

—

Project

<a href="/en/project/FR-TI1%2F486" target="_blank" >FR-TI1/486: *MegaWord.cz</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Publication year

2011

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Lecture Notes in Computer Science

ISSN

0302-9743

e-ISSN

—

Volume of the periodical

Neuveden

Issue of the periodical within the volume

6836

Country of publishing house

DE - GERMANY

Number of pages

8

Pages from-to

356-363

UT code for WoS article

—

EID of the result in the Scopus database

—