1.34.21

1.34.21 (2025-02-18)

Fix

• fix: Add back task filtering by modalities (#2080)

• add back task filtering by modalities

• add unit test

• check if task modalities is a subset of model modalities and fix tests

• add model_modalities_more_than_task_modalities case (3deb7ea)

Unknown

• merge gme models (#2089) (1b1d327)

1.34.20

1.34.20 (2025-02-17)

Fix

• fix: Missing fixes for #2086 - change MultilingualSentiment split from test to validation in CMTEB (#2088)

• fix: Fixed previous incorrect specification of splits for CMTEB ( MTEB(cmn, v1) )

Fixes #2064

• change MultilingualSentiment split from test to validation in CMTEB (6637ff9)

• fix: Smarter leaderboard caching with cachetools (#2085)

• Added smarter caching to callbacks

• Added cachetools as a dependency

• Ran linting

• Removed debugging print statement

• Bumped Gradio version

• Dependency fixes

• Dependency fixes

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (1006770)

Unknown

• Remove duplicated string in docstring of TaskMetadata class (#2087)

• Remove duplicated string in docstring of TaskMetadata class

• Remove duplicated dataset field (c6e5123)

1.34.19

1.34.19 (2025-02-17)

Fix

• fix: Fixed previous incorrect specification of splits for CMTEB ( MTEB(cmn, v1) ) (#2086)

Fixes #2064 (12d9b96)

1.34.18

1.34.18 (2025-02-17)

Fix

• fix: Freeze model/rank columns in leaderboard (#2044)

• fix: freeze model/rank columns in leaderboard

• freezing zero-shot column

• update min gradio version to 5.16.0 in pyproject.toml

---

Co-authored-by: Shikhar Shiromani <[email protected]> (07562f4)

1.34.17

1.34.17 (2025-02-17)

Fix

• fix: Ensure voyage model uses different naming scheme (#2083)

• fix: Added make command for running leaderboard locally

• fix: Ensure voyage models doesn't re-use the name (b14963f)

1.34.16

1.34.16 (2025-02-17)

Fix

• fix: add missing e5 training datasets (#2065)

add missing training datasets (efe2578)

1.34.15

1.34.15 (2025-02-17)

Ci

• ci: Rerun tests that fail due to networking issues. (#2029)

• fix: rerun tests that fail - Networking

• update tests to use tmp_path

• set versions for dev dependencies

• add pytest options to pyproject.toml

• add rerun json.decoder.JSONDecodeError

• remove JSONDecodeError from pyproject.toml

• add huggingface_hub.errors.HfHubHTTPError

• add huggingface_hub.errors.LocalEntryNotFoundError
  https://github.com/embeddings-benchmark/mteb/actions/runs/13298535701/job/37139767443?pr=2044

• FileNotFoundError
  https://github.com/embeddings-benchmark/mteb/actions/runs/13302915091/job/37147507251?pr=2029

• add doc to pytest rerun

---

Co-authored-by: sam021313 <[email protected]> (efaa990)

Fix

• fix: generate metadata (#2063)

• fix: generate metadata

• use logging not print for script

• lint

• add iso639 to dev pyproject

• fix import

• add memory_usage_mb

• set version for iso639

Co-authored-by: Kenneth Enevoldsen <[email protected]>

---

Co-authored-by: sam021313 <[email protected]>
Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Roman Solomatin <[email protected]> (26360a0)

Unknown

• Update tasks table (3dbdeb1)

• misc: Add all image classification descriptive stats (#2073)

• add most image classification descr stats

• revert changes to encoder

• add stats

---

Co-authored-by: Roman Solomatin <[email protected]> (04c9993)

• misc: update model names to adjust for adding to results repo (#2074)

• update model names to adjust for adding to results repo

• update model meta script (50cc1c9)

• Update tasks table (dbda3c5)

• Add datasets for a benchmark newly introduced for "Engineering" domain (#1911)

• adding clustering tasks (built-bench-clustering S2S & P2P)

• updated built-bench-clustering tasks

• Updated BuiltBenchClustering tasks

• Added "Engineering" as new domain to TaskMetadata.py

• Updated tasks table in docs

• Updated task metadata for BuiltBenchClustering S2S and P2P

• updated metadata for clustering tasks

• Add/update BuiltBench tasks

• Add BuiltBenchRetrieval task
• Add BuiltBenchReranking task
• Update metadata for BuiltBenchClusterinP2P
• Update metadata for BuiltBenchClusterinS2S

• update BuiltBench benchmark

• Update mteb/benchmarks/benchmarks.py

Co-authored-by: Roman Solomatin <[email protected]>

• Update mteb/tasks/Clustering/eng/BuiltBenchClusteringS2S.py

Co-authored-by: Roman Solomatin <[email protected]>

• Update mteb/tasks/Clustering/eng/BuiltBenchClusteringP2P.py

Co-authored-by: Roman Solomatin <[email protected]>

• Update mteb/benchmarks/benchmarks.py

Co-authored-by: Isaac Chung <[email protected]>

• Fix formatting via ruff

---

Co-authored-by: Roman Solomatin <[email protected]>
Co-authored-by: Isaac Chung <[email protected]> (5f4b593)

1.34.14

1.34.14 (2025-02-14)

Fix

• fix: Updating paper scripts (#1958)

• change reference revisions to align with paper

• Update author list

• Added code for main results table

• updated minor changes

• added external as a "no_revision_available" case

• revert unintended changes

• format (c6829d3)

• fix: Add climate fever v2 (#1873)

• Updated ClimateFEVER dataset with new version

• Adds Fill in the empty metadata.

• Updates the date tuple

• Update class name

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update domains

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update task_subtypes

• Update annotations_creators for the first version

• Update date

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• Update task subtypes

• Update path

• Update description

---

Co-authored-by: Kenneth Enevoldsen <[email protected]>
Co-authored-by: Mina Parham <[email protected]> (8604e07)

• fix: Added gte models (#1539)

• fix: Added gte models

• fix: Add mixbai models (#1540)

for #1515 (76e05dd)

Unknown

• Update tasks table (11ced79)

• Update tasks table (479fa20)

• misc: Add VisualSTS descriptive stats (#2062)

• add visualsts stats

• add last dataset (3cf7b15)

• Rename MIEB task classes with duplicated names (#2061)

fix class names (bef4046)

• Update tasks table (e090330)

• misc: Add ZS and multilabel image classification descriptive stats implementation (#2059)

• add image clustering descirptive stats and run

• finish off last one

• remove script

• add ImageMultilabelClassificationDescriptiveStatistics

• add VOC2007

• add zeroshot and mnist example (20df284)

• Add giga embeddings (#1741)

• add gigaembeddings

• use jasper

• fix name

• create sentence_transformer instruct wrapper

• apply instruction template

• fix jasper

• update meta (02d2583)

1.34.13

1.34.13 (2025-02-13)

Fix

• fix: Update embed_dim for jina models (#2058)

see embeddings-benchmark/results#117 (50b8e7b)

Unknown

• Update tasks table (48ef6f4)

• misc: Add image clustering descriptive stats implementation (#2057)

• add image clustering descirptive stats and run

• finish off last one

• remove script (eb32719)

1.34.12

1.34.12 (2025-02-13)

Fix

• fix: Add BRIGHT (long) and fix bug in TaskResult.filter_and_validate() (#2041)

• fix: Add BRIGHT Long

Fixes #1978

• fix: Add BRIGHT(long)

• fix bug in task results

• updated bright

• updated tests for TaskResults (3537223)

• fix: Add column descriptions to leaderboard (#2039)

• fix: Add column descriptions to leaderboard

• removed existing overlap (01fd6fb)

Unknown

• Update tasks table (fadba48)

• misc: Add image classification descriptive stats implementation (#2045)

• add ImageClassificationDescriptiveStatistics

• add MNIST descriptive stats

• use tuples instead

• add label count and update docstrings

• update MNIST example (9556f99)