N-Gram-Based Text Compression

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989100%3A27240%2F16%3A86099098" target="_blank" >RIV/61989100:27240/16:86099098 - isvavai.cz</a>
Result on the web
<a href="http://downloads.hindawi.com/journals/cin/2016/9483646.pdf" target="_blank" >http://downloads.hindawi.com/journals/cin/2016/9483646.pdf</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1155/2016/9483646" target="_blank" >10.1155/2016/9483646</a>

Result language
angličtina
Original language name
N-Gram-Based Text Compression
Original language description
We propose an efficient method for compressing Vietnamese text using n-gram dictionaries. It has a significant compression ratio in comparison with those of state-of-the-art methods on the same dataset. Given a text, first, the proposed method splits it into n-grams and then encodes them based on n-gram dictionaries. In the encoding phase, we use a sliding window with a size that ranges from bigram to five grams to obtain the best encoding stream. Each n-gram is encoded by two to four bytes accordingly based on its corresponding n-gram dictionary. We collected 2.5 GB text corpus from some Vietnamese news agencies to build n-gram dictionaries from unigram to five grams and achieve dictionaries with a size of 12 GB in total. In order to evaluate our method, we collected a testing set of 10 different text files with different sizes. The experimental results indicate that our method achieves compression ratio around 90% and outperforms state-of-the-art methods. (C) 2016 Vu H. Nguyen et al.
Czech name
—
Czech description
—

Type
J<sub>x</sub> - Unclassified - Peer-reviewed scientific article (Jimp, Jsc and Jost)
CEP classification
IN - Informatics
OECD FORD branch
—

Publication year
2016
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Similar results(10)