1.34.14

1.34.14 (2025-02-14)

Fix

• fix: Updating paper scripts (#1958)

• change reference revisions to align with paper

• Update author list

• Added code for main results table

• updated minor changes

• added external as a "no_revision_available" case

• revert unintended changes

• format (c6829d3)

• fix: Add climate fever v2 (#1873)

• Updated ClimateFEVER dataset with new version

• Adds Fill in the empty metadata.

• Updates the date tuple

• Update class name

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update domains

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update task_subtypes

• Update annotations_creators for the first version

• Update date

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update task subtypes

• Update path

• Update description

---

Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Mina Parham <[email protected]> (8604e07)

• fix: Added gte models (#1539)

• fix: Added gte models

• fix: Add mixbai models (#1540)

for #1515 (76e05dd)

Unknown

• Update tasks table (11ced79)

• Update tasks table (479fa20)

• misc: Add VisualSTS descriptive stats (#2062)

• add visualsts stats

• add last dataset (3cf7b15)

• Rename MIEB task classes with duplicated names (#2061)

fix class names (bef4046)

• Update tasks table (e090330)

• misc: Add ZS and multilabel image classification descriptive stats implementation (#2059)

• add image clustering descirptive stats and run

• finish off last one

• remove script

• add ImageMultilabelClassificationDescriptiveStatistics

• add VOC2007

• add zeroshot and mnist example (20df284)

• Add giga embeddings (#1741)

• add gigaembeddings

• use jasper

• fix name

• create sentence_transformer instruct wrapper

• apply instruction template

• fix jasper

• update meta (02d2583)

1.34.13

1.34.13 (2025-02-13)

Fix

• fix: Update embed_dim for jina models (#2058)

see embeddings-benchmark/results#117 (50b8e7b)

Unknown

• Update tasks table (48ef6f4)

• misc: Add image clustering descriptive stats implementation (#2057)

• add image clustering descirptive stats and run

• finish off last one

• remove script (eb32719)

1.34.12

1.34.12 (2025-02-13)

Fix

• fix: Add BRIGHT (long) and fix bug in TaskResult.filter_and_validate() (#2041)

• fix: Add BRIGHT Long

Fixes #1978

• fix: Add BRIGHT(long)

• fix bug in task results

• updated bright

• updated tests for TaskResults (3537223)

• fix: Add column descriptions to leaderboard (#2039)

• fix: Add column descriptions to leaderboard

• removed existing overlap (01fd6fb)

Unknown

• Update tasks table (fadba48)

• misc: Add image classification descriptive stats implementation (#2045)

• add ImageClassificationDescriptiveStatistics

• add MNIST descriptive stats

• use tuples instead

• add label count and update docstrings

• update MNIST example (9556f99)

1.34.11

1.34.11 (2025-02-12)

Fix

• fix: add Voyage-code-3 (#2040) (0b37966)

1.34.10

1.34.10 (2025-02-12)

Documentation

• docs: Fix README code rendering (#2037)

Fix README code rendering (1b04130)

Fix

• fix: Add versioning to MTEB benchmarks (#2024)

• Add versioning to MTEB benchmarks

• Following suggestion made in #2001 I added version to MTEB benchmarks
• changed the name of the MTEB(Chinese) to MTEB(cmn, v1). Though we could go for MTEB(Chinese, v1) assuming it is a group (also covering chinese other than mandarin)
• change the name of FaMTEB(fas, beta) to MTEB(fas)
• did a minor formatting of imports due to a circular import error
• moved the Benchmark object out of the file with the benchmarks
  • this is still >1000 lines so we could split it up to "external_benchmarks", "monolingual_benchmarks", "domain_specific_benchmarks", "multilingual_benchmarks"

Fixes #2001

• format

• update tests

• updated for backward compatibility

• ibid

• Update mteb/benchmarks/get_benchmark.py

Co-authored-by: Roman Solomatin <[email protected]>

• Update mteb/benchmarks/get_benchmark.py

Co-authored-by: Roman Solomatin <[email protected]>

• add missing import

---

Co-authored-by: Roman Solomatin <[email protected]> (65f3407)

1.34.9

1.34.9 (2025-02-12)

Fix

• fix: Add SONAR metadata and resolve missing models (#2014)

• Add SONAR metadata

Add SONAR metadat, but without an implementation

Fixes #1981

• fix: Add SONAR metadata

Fixes #1981

• minor edits

• reduced logging serverity of no model_meta.json

• resolve missing models

by ensuring that "Unknown" number of parameters is not filtered.

Should resolve:
#1979
#1976

This seems to have been caused by the restructering of calls on the leaderboard.

• format

• resolve missing models

by ensuring that "Unknown" number of parameters is not filtered.

Should resolve:
#1979
#1976

This seems to have been caused by the MAX_MODEL_SIZE/MIN_MODEL_SIZE args.

• format

• format

• added memory usage

• fixed None checks

• consistently refer to tasks as tasks not as datasets

adresses #2026

• minor

• removed used arg

• revert fix of not allowing None in model name (92b74b6)

• fix: Added script for generating and saving a local leaderboard (#2015)

• Added script for making a local leaderboard

• Updated variable

• Update scripts/make_leaderboard.py

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Resolved comments

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (477bea5)

Unknown

• Add SFR-Embedding-Code-2B_R (#2032)

• Add SFR-Embedding-Code-2B_R(#2019)

• make lint

• Added HF repo name for adapted_from parameter

• replace instruct_wrapper (0caea67)

• Add FRIDA model meta (#2031)

• add FRIDA to models

• update prompts (f3c71b8)

1.34.8

1.34.8 (2025-02-10)

Documentation

• docs: update MTEB eng classic benchmark description (#2006)

• Simplify text

• Clarify

Co-authored-by: Kenneth Enevoldsen <[email protected]>

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (7917646)

• docs: ModelMeta docstrings Typos (#2017)

• Update model_meta.py

• Update mteb/model_meta.py

---

Co-authored-by: Isaac Chung <[email protected]> (42cf6a0)

• docs: Update adding_a_model.md (#2018)

Update adding_a_model.md (e6e21dc)

Fix

• fix: Add Persian-Specific Models (#2021)

• Add new models

• Add training datasets info

---

Co-authored-by: mehran <[email protected]> (1588b9a)

Unknown

• Adding sentence-transformer-xxl model to mteb (#2023)

• add sentence transformer xxl

• update datasets

• make lint, and rm public training code

• add all sentence-transformer models (f7a81e3)

• add model memory usage (#1934)

• add model memory usage

• lint

• update

• calculate memory usage based on file size

• calculate memory usage

• add memory usage for MIEB models

• add last model usage

• add memory_usage_mb to overview

• fix rerank

• update memory usage

• update memory usage (e46539a)

1.34.7

1.34.7 (2025-02-07)

Fix

• fix: BEIR-NL metadata mistake (#2010)

Update SCIDOCSNLRetrieval.py (b1ac052)

Unknown

• Update tasks table (64f256e)

1.34.6

1.34.6 (2025-02-07)

Fix

• fix: Update faq of on leaderboard (#2004)

Simplify (4fe4c99)

1.34.5

1.34.5 (2025-02-07)

Fix

• fix: training data for gritlm (#1932)

• Fix training data

• Fix data

• Fix data

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (d810e4e)

Unknown

• Sort benchmarks and add temp. reference to old leaderboard (#1993)

• fix: Added description and resolved bug in rangeslider

addresses #1987
fixes #1954

• format

• remove certificate

• Sorted benchmarks #1984 (3887d83)

• Add gte-modernbert-base (#1845)

• Update gte_models.py

• Update gte_models train data

• Update gte_models.py

• Update gte_models.py

• Update gte_models.py

• Update gte_models.py

• Update mteb/models/gte_models.py

• Update gte_models.py

• Update gte_models.py

• Update gte_models.py

---

Co-authored-by: Roman Solomatin <[email protected]> (e2c44ed)

• CI clean up: Remove MMTEB points check (#1994)

Fixes #1992 (fb6bade)