Application of Lemmatization and Summarization Methods in Topic Identification Module for Large Scale Language Modeling Data Filtering

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F12%3A43915504" target="_blank" >RIV/49777513:23520/12:43915504 - isvavai.cz</a>

Result on the web

<a href="http://link.springer.com/chapter/10.1007%2F978-3-642-32790-2_23" target="_blank" >http://link.springer.com/chapter/10.1007%2F978-3-642-32790-2_23</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-642-32790-2_23" target="_blank" >10.1007/978-3-642-32790-2_23</a>

Result language

angličtina

Original language name

Application of Lemmatization and Summarization Methods in Topic Identification Module for Large Scale Language Modeling Data Filtering

Original language description

The paper presents experiments with the topic identification module which is a part of a complex system for acquisition and storing large volumes of text data. The topic identification module processes each acquired data item and assigns it topics from adefined topic hierarchy. The topic hierarchy is quite extensive - it contains about 450 topics and topic categories. It can easily happen that for some narrowly focused topic there is not enough data for the topic identification training. Lemmatizationis shown to improve the results when dealing with sparse data in the area of information retrieval, therefore the effects of lemmatization on topic identification results is studied in the paper. On the other hand, since the system is used for processinglarge amounts of data, a summarization method was implemented and the effect of using only the summary of an article on the topic identification accuracy is studied.

Czech name

—

Czech description

—

Type

J_x - Unclassified - Peer-reviewed scientific article (Jimp, Jsc and Jost)

CEP classification

JD - Use of computers, robotics and its application

OECD FORD branch

—

Project

<a href="/en/project/LM2010013" target="_blank" >LM2010013: LINDAT-CLARIN: Institute for analysis, processing and distribution of linguistic data</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Publication year

2012

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Lecture Notes in Computer Science

ISSN

0302-9743

e-ISSN

—

Volume of the periodical

7499

Issue of the periodical within the volume

Neuveden

Country of publishing house

DE - GERMANY

Number of pages

8

Pages from-to

191-198

UT code for WoS article

—

EID of the result in the Scopus database

—