HSEARCH-5052 Filter out documents with vectors below a "similarity limit"

[marko-bekhta]

https://hibernate.atlassian.net/browse/HSEARCH-5052

Hey Yoann 😃
You've probably seen the discussion we had yesterday on Zulip here: https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/topic/vector.20search/near/407994976
I thought I'd give it a try and see what can we do to filter things out on our side; I've looked into creating custom queries and that's what I've put together. Do you think this could be something useful to add?

[hibernate-github-bot]

Thanks for your pull request!

This pull request appears to follow the contribution rules.

› This message was automatically generated.

[yrodiere]

Do you think this could be something useful to add?

For casual users, I very much doubt it's useful.

For advanced users... I can't say, I'm not familiar enough with the topic. I'm not sure the "similarity" value is meaningful enough for users to even be able to provide a meaningful limit, but then... I have little experience with this.

If you can find good use cases, and it's not too hard to support on Elasticsearch/OpenSearch, well... why not. But we'd have to be extra clear this is for very specific use cases.

[yrodiere]

If you can find good use cases, and it's not too hard to support on Elasticsearch/OpenSearch, well... why not. But we'd have to be extra clear this is for very specific use cases.

Ok, this gives a rather interesting use case: https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html#knn-similarity-search

Basically it's not about relevance, it's about performance. A similarity filter stops the search early if the engine can't find any vector that is reasonably similar to the provided one, avoiding unnecessary computation. In other words, we know that vectors that have a similarity lower than a given number won't affect the score that much, so we don't even look for them.

So yes, I think you're right, this is a valuable feature. Please create a Jira?

[yrodiere]

Thanks! LGTM, just one comment.

[sonarqubecloud]

Quality Gate failed

Failed conditions

65.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud