Changes related to Meilisearch v1.13.0

[curquiza]

This issue gathers the changes related to the v1.13.0 of Meilisearch that will impact the integrations scope.

📅 Release date: 17th February

Timelines & steps

Pre-release

• With the help of the changelog & this CI, define which integrations should be updated and how (New feature? Update README? Update tests?)
  • Fill in the "What to implement?" section below in this issue 👇. Minial implementation: PHP, JS, meilisearch-js-plugins.
  • Open implementation issues in repositories that need implementation if necessary for developers who will work on the tasks
  • Discuss with the Product team if needed. At least share this issue to let them know about the decisions.
• Create a branch by running Octopus script: only open branches for the integrations we choose to update (defined in the previous step) + Kubernetes repository + Cloud provider repository (changing the version)
• Update integrations according to the decisions (cf "What to implement?" section below in this issue 👇)
  ⚠️ If possible, this step is done before pre-release, once the feature is ready thanks to the prototype released by the engine team
  • JS
  • PHP
  • Dart
  • Go
  • Java
  • Python
  • Ruby
  • Rust
• Add code samples for the chosen up-to-date integrations with the new version of Meilisearch
• Update the library version of the related integrations and prepare the changelogs

Release day

• Release the integrations or only merge the related PRs (sometimes a release is not needed)
  • JS v0.49
  • PHP v1.13
  • meilisearch-js-plugins (at least update meilisearch-js version)
  • Dart v0.17
  • Go v0.31
  • Python v0.34
  • Rust v0.28
• Merge the related PR in K8s repository
• Publish DevOps tools:
  • create the git tag
  • publish images (steps are in CONTRIBUTING.md)
• Remove useless AWS images (by using our internal script)
• Open issues in the repositories that are not up-to-date with the latest version of Meilisearch (including code samples)

Post-release

• Open issues post-release for stabilization of AI
• Update the following SDKs with the missing capabilities for AI search
  • Dart: TBD
  • Go: TBD
  • Java: TBD
  • Python: TBD
  • Ruby: TBD
  • Rust: TBD
• Open issues in Swift and Dotnet for the missing AI capabilities

What to implement?

AI stabilization

Related issue in the engine: meilisearch/meilisearch#4733

Usage: https://meilisearch.notion.site/v1-13-AI-search-changes-17a4b06b651f80538b65d31724545def

• removal of the vectorStore experimental feature
• new error codes

TODO:

• JS: remove vectorStore exp feature + fix tests
• PHP: remove vectorStore exp feature + fix tests
• Dart: Remove vector store experimental feature · Issue #409 · meilisearch/meilisearch-dart Why are tests not breaking?
• Go: Remove vector store experimental feature · Issue #605 · meilisearch/meilisearch-go
• Java: Remove vector store experimental feature · Issue #812 · meilisearch/meilisearch-java
• Python: Remove vector store experimental feature · Issue #1077 · meilisearch/meilisearch-python
• Ruby: Remove vector store experimental feature · Issue #605 · meilisearch/meilisearch-ruby
• Rust: Remove vector store experimental feature · Issue #644 · meilisearch/meilisearch-rust

Postponed to v1.14.0
### Granular filterable attributes

Related issue in the engine: meilisearch/meilisearch#5163

Usage: https://meilisearch.notion.site/API-usage-Settings-to-opt-out-indexing-features-filterableAttributes-1764b06b651f80aba8bdf359b2df3ca8?pvs=74

• accept the new formal of filterable attributes (the old one is still allowed)

TODO:

• JS: adapt code base + fix tests
• [ ] PHP: adapt code base + fix tests

Post release day - implement missing feature for vector store

Spec

• embedders setting. Methods getEmbedders, updateEmbedders, resetEmbedders. Also, the method updateSettings should be able to accept the new embedders field. Here is the list of the acceptable sub fields:
  • source sub field is available and accepts: ollama, rest, openAI, huggingFace and userProvided
  • apiKey sub field is available (string) - optional because not compatible with all sources. Only for openAi, ollama, rest.
  • model sub field is available (string) - optional because not compatible with all sources. Only for ollama, openAI, huggingFace
  • documentTemplate sub field is available (string) - optional
  • dimensions - optional because not compatible with all sources. Only for openAi, huggingFace, ollama, and rest
  • distribution - optional
  • request - mandatory only if using rest embedder
  • response - mandatory only if using rest embedder
  • documentTemplateMaxBytes - optional
  • revision - optional, only for huggingFace
  • headers - optional, only for rest
  • binaryQuantized - optional
• Search
  • hybrid search parameter, with sub fields semanticRatio and embedder. embedder is mandatory if hybrid is set.
  • vector parameter is available
  • retrieveVectors parameter available
  • semanticHitCount in search response
  • Accept _semanticScore in the search response (optional)
  • vector should be returned in the search response, but optional (because depends on search parameters)
  • _vectors should NOT be present in the search response
• Similar. Implement searchSimilarDocuments associated with the POST /indexes/:uid/similar. Do NOT implement with GET.

TODO

• Dart: TBD
• Go: TBD
• Java: TBD
• Python: TBD
• Ruby: TBD
• Rust: TBD

Post-release: Remote federated search requests (exp)

Related issue in the engine: meilisearch/meilisearch#4980

Usage: https://meilisearch.notion.site/API-usage-Remote-search-request-f64fae093abf409e9434c9b9c8fab6f3

• New remote route
• accept remote search parameter in /multi-search route

TODO:

• JS: adapt code base + fix tests
• PHP: adapt code base + fix tests

Post-release: New usedDatabaseSize in /stats

Usage: meilisearch/meilisearch#5318

• Accept the usedDatabaseSize in the response of the getStats method

TODO

• JS
• PHP: no need to change 🎉

Issue to open after the release: .NET, dart, go, Java, Rust, Swift (not Ruby, or Python)

Post-release: Add embbedings database metrics to stats route

Usage: meilisearch/meilisearch#5321

• Accept indexes.INDEXNAME.numberOfEmbeddedDocuments in the response of the getStats method
• Accept indexes.INDEXNAME.numberOfEmbeddings in the response of the getStats method

TODO

• JS
• PHP: no need to change 🎉

Post-release: Add documents database metrics to stats route

meilisearch/meilisearch#5319

• Accept indexes.INDEXNAME.rawDocumentDbSize in the response of the getStats method
• Accept indexes.INDEXNAME.maxDocumentSize in the response of the getStats method
• Accept indexes.INDEXNAME.avgDocumentSize in the response of the getStats method

TODO

• JS
• PHP: no need to change 🎉

[ellnix]

Granular filterable attributes
Related issue in the engine: meilisearch/meilisearch#5163
Usage: https://meilisearch.notion.site/Opt-out-settings-v2-usage-15c4b06b651f8188ae57fca7aa73b527

accept the new formal of filterable attributes (the old one is still allowed)

@curquiza I don't think that notion link is correct.

[curquiza]

Thank you, I changed

[curquiza]

@Strift, the Granular filterable attributes feature will not be ready for v1.13.0, but will be done in v1.14.0
Just got the new from engine team, who had difficulties managing the flood of bugs and the sprint lately

[CommanderStorm]

How do we want to go about the rust integration?
(I have not updated my PR to remove the experimental part, but can do so)

Kontext:

• Support embedders setting meilisearch-rust#554

These are the options

• remove ExperimentalFeatures::set_vector_store
• mark ExperimentalFeatures::set_vector_store as deprecated and change to a no-op

And about reviewability:

• include in the same PR
• make a new PR

[ellnix]

There seems to be precedent for this @CommanderStorm

meilisearch/meilisearch-rust@d12b21b

meilisearch/meilisearch-rust#544

Looks like it was just removed (I suppose it was "experimental" after all). You know better though, of course doing something different could be appropriate in your case. Just thought I'd share.