SlamaTrain – Representative Training Dataset for Slavonic Large Language Models

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216224%3A14330%2F24%3A00138085" target="_blank" >RIV/00216224:14330/24:00138085 - isvavai.cz</a>

Výsledek na webu

—

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

SlamaTrain – Representative Training Dataset for Slavonic Large Language Models

Popis výsledku v původním jazyce

The Slama project focuses on building a series of foundational language models for Slavonic languages. Even though the latest developmentyieldsanumberofnewlargepre-trainedandfine-tunedmodels,the main data source came from English-written websites. Therefore the majority of the training data that is used for language model development consists oftheEnglishlanguage.MultilinguallanguagemodelslikeLlama, GPT-4o,mT5,etc.arealsopredominantly(around80%)trainedontheEnglish language, even though they capture the structure of dozens of languages. In this paper, we detail the process of acquiring one of the largest training datasets for Czech, Slovak and other Slavonic languages. We started with huge multi-lingual datasets, extracted the mono-lingual data and joined them with other sources. The combined mono-lingual datasets were then cleaned, deduplicated and filtered for adult content. As a result, we have obtained 71 billion tokens for the Czech and Slovak languages suitable for the Slama language models training.

Název v anglickém jazyce

SlamaTrain – Representative Training Dataset for Slavonic Large Language Models

Popis výsledku anglicky

The Slama project focuses on building a series of foundational language models for Slavonic languages. Even though the latest developmentyieldsanumberofnewlargepre-trainedandfine-tunedmodels,the main data source came from English-written websites. Therefore the majority of the training data that is used for language model development consists oftheEnglishlanguage.MultilinguallanguagemodelslikeLlama, GPT-4o,mT5,etc.arealsopredominantly(around80%)trainedontheEnglish language, even though they capture the structure of dozens of languages. In this paper, we detail the process of acquiring one of the largest training datasets for Czech, Slovak and other Slavonic languages. We started with huge multi-lingual datasets, extracted the mono-lingual data and joined them with other sources. The combined mono-lingual datasets were then cleaned, deduplicated and filtered for adult content. As a result, we have obtained 71 billion tokens for the Czech and Slovak languages suitable for the Slama language models training.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10200 - Computer and information sciences

Projekt

<a href="/cs/project/LM2023062" target="_blank" >LM2023062: Digitální výzkumná infrastruktura pro jazykové technologie, umění a humanitní vědy</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Recent Advances in Slavonic Natural Language Processing, RASLAN 2024

ISBN

9788026318354

ISSN

2336-4289

e-ISSN

—

Počet stran výsledku

9

Strana od-do

25-33

Název nakladatele

Tribun EU

Místo vydání

Brno, Czech Republic

Místo konání akce

Kouty nad Desnou, Česká Republika

Datum konání akce

1. 1. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—