Distributed Training for Multilingual Combined Tokenizer using Deep Learning Model and Simple Communication Protocol

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F19%3A10427046" target="_blank" >RIV/00216208:11320/19:10427046 - isvavai.cz</a>

Result on the web

—

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

Distributed Training for Multilingual Combined Tokenizer using Deep Learning Model and Simple Communication Protocol

Original language description

In the big data era, text processing tends to be harder as the data increase. There is also the growth of deep learning model for solving natural language processing tasks without a need for hand-crafted rules. In this research, we provide two big solutions in the area of text preprocessing and distributed training for any neural-based model. We try to solve the most common text preprocessing which are word and sentence tokenization. Our proposed combined tokenizer is compared by using a single language model and multilanguage model. We also provide a simple communication using MQTT protocol to help the training distribution.

Czech name

—

Czech description

—

Type

O - Miscellaneous

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2019

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů