An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A10511576" target="_blank" >RIV/00216208:11320/25:10511576 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2025.acl-long.854/" target="_blank" >https://aclanthology.org/2025.acl-long.854/</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.18653/v1/2025.acl-long.854" target="_blank" >10.18653/v1/2025.acl-long.854</a>
Alternative languages
Result language
angličtina
Original language name
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
Original language description
Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual, monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2025
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
ISBN
979-8-89176-251-0
ISSN
—
e-ISSN
—
Number of pages
34
Pages from-to
17452-17485
Publisher name
Association for Computational Linguistics
Place of publication
Kerrville, TX, USA
Event location
Wien, Austria
Event date
Jul 27, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—