Add vector search documentation

[kolchfa-aws]

Adds a vector search section

Checklist

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

[naveentatikonda]

Faiss HNSW with PQ also requires training

[naveentatikonda]

Need to add a card for binary vectors along with byte vectors
https://opensearch.org/docs/latest/field-types/supported-field-types/knn-vector#binary-vectors

[kolchfa-aws]

@naveentatikonda Thanks! I addressed both comments. Could you review this commit a5e8b8d.

[naveentatikonda]

@kolchfa-aws changes looks good. Thanks for making those changes.

For binary vectors, we also need to add memory estimation. Here is the formula for HNSW
1.1 * (dimension / 8 + 8 * M) bytes/vector

For IVF, I guess it is 1.1 * (((dimension / 8) * num_vectors) + (nlist * dimension / 8)). @jmazanec15 can you pls confirm ?

[jmazanec15]

Yeah that looks good to me

[jmazanec15]

Hi @kolchfa-aws, this looks awesome!

In general, I think we should start moving the more low-level/expert details (like quantization and method configuration), out of vector search section and into detailed field reference section. Here is some high level feedback:

1. Itd be good to showcase/highlight some high level features on the vector search splash page (https://kolchfa-aws.github.io/vector-search). In particular: filtering, multi-vector per document support (nested), automatic embedding generation, low-memory search, and hybrid search
2. For vector search techniques, can we add sections on sparse vectors and hybrid search?
3. Can we add knn query type in query dsl?
4. Can we have a page for space types and only maintain one table - possibly in field reference? Also, can we move engine/method details into field reference and each engine have its own detailed section in mapping reference.
5. For examples/tutorials, can we use basic mapping instead of complex mapping:

PUT /test-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector1": {
        "type": "knn_vector",
        "dimension": 3,
        "space_type": "l2"
      }
    }
  }
}

In performance tuning, we can mention picking a specific engine or specifying overriding method parameters for expert level fine tuning and point to reference docs.
6. For quantization, can we put low level details into an expert section? Instead, can we just mention in performance tuning for memory optimization, we use quantization to achieve 32x, 16x, 8x, 4x, 2x compression_levels. I think we can even renaming the disk-based section, memory optimized vector search. This is example mapping:

PUT my-vector-index
{
  "settings" : {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector_field": {
        "type": "knn_vector",
        "dimension": 8,
        "space_type": "innerproduct",
        "data_type": "float",
        "mode": "on_disk",
        "compression_level": "16x"
      }
    }
  }
}

Quantization is a bit ugly for users to have to understand. So I think its better to belong in detailed field reference. We can say, for further fine tuning of the quantization methods, see field reference.
7. Can we move SIMD optimization section to performance tuning: https://kolchfa-aws.github.io/vector-search/creating-vector-index/vector-field/#simd-optimization-for-the-faiss-engine? End users dont really need to know about this when reading about vector data types
8. In optimizing vector search performance, can we add page on cluster sizing? Basically, this should point to memory evaluation formulas and picking node configurations.