CUNI at WMT24 General Translation Task: LLMs, (Q)LoRA, CPO and Model Merging
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F24%3A10492861" target="_blank" >RIV/00216208:11320/24:10492861 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2024.wmt-1.16/" target="_blank" >https://aclanthology.org/2024.wmt-1.16/</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.18653/v1/2024.wmt-1.16" target="_blank" >10.18653/v1/2024.wmt-1.16</a>
Alternative languages
Result language
angličtina
Original language name
CUNI at WMT24 General Translation Task: LLMs, (Q)LoRA, CPO and Model Merging
Original language description
This paper presents the contributions of Charles University teams to the WMT24 General Translation task (English to Czech, German and Russian, and Czech to Ukrainian) and the WMT24 Translation into Low-Resource Languages of Spain task. Our most elaborate submission, CUNI-MH for en2cs, is the result of fine-tuning Mistral 7B v0.1 for translation using a three-stage process: Supervised fine-tuning using QLoRA, Contrastive Preference Optimization, and merging of model checkpoints. We also describe the CUNI-GA, CUNI-Transformer and CUNI-DocTransformer submissions, which are based on our systems from the previous year. Our en2ru system CUNI-DS uses a similar first stage as CUNI-MH (QLoRA for en2cs) and follows with transfer learning for en2ru. For en2de (CUNI-NL), we experimented with an LLM-based speech translation system, to translate without the speech input. For the Translation into Low-Resource Languages of Spain task, we performed QLoRA fine-tuning of a large LLM on a small amount of synthetic (backt
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the Ninth Conference on Machine Translation
ISBN
979-8-89176-179-7
ISSN
—
e-ISSN
—
Number of pages
15
Pages from-to
232-246
Publisher name
Association for Computational Linguistics
Place of publication
Kerrville, TX, USA
Event location
Miami, FL, USA
Event date
Nov 15, 2024
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—