MMTEB: MASSIVE MULTILINGUAL TEXT EMBEDDING BENCHMARK

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F26%3AP6RH5DK4" target="_blank" >RIV/00216208:11320/26:P6RH5DK4 - isvavai.cz</a>

Result on the web

—

DOI - Digital Object Identifier

—

Result language

angličtina

Original language name

MMTEB: MASSIVE MULTILINGUAL TEXT EMBEDDING BENCHMARK

Original language description

Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a similar ranking order as the full-scale version but only requires 2% of the original documents vastly reducing the computational cost. © 2025 13th International Conference on Learning Representations, ICLR 2025. All rights reserved.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

—

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Int. Conf. Learn. Represent., ICLR

ISBN

979-8-3313-2085-0

ISSN

—

e-ISSN

—

Number of pages

57

Pages from-to

102004-102060

Publisher name

International Conference on Learning Representations, ICLR

Place of publication

—

Event location

Singapore

Event date

Jan 1, 2026

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—