Increment version to 2.20.0-SNAPSHOT #2522
Closed
+88,531
−16,353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a set of JNI functions that allows IVFPQ-l2 dynamically precomputed tables to be shared amongst indices at load time that use the same model. In addition, refactors JNIService interface to take KNNEngine as param instead of as string In addition, add unit tests to confirm that the shared state operates as expected. Signed-off-by: John Mazanec <[email protected]>
…#1545) Adds the ability to share index state amongst indices during index load operations into the plugins memory management system. Introduces a manager of the shared state that will properly manage the lifecycle of the shared state. There was a bug in clear cache that had to be fixed to get this change working as well. Previously, only one index file per clear cache would be freed. This fixes that logic to clear everything. Added unit tests and an integration test to confirm functionality. In addition, modified recall integration tests to get more coverage on the different algo configs. Along with this, had to fix a few things around the computation of recall for non-l2 space types. Signed-off-by: John Mazanec <[email protected]>
) Signed-off-by: Naveen Tatikonda <[email protected]> (cherry picked from commit 4b0078d) Co-authored-by: Naveen Tatikonda <[email protected]>
…-project#1563) * Faiss SQFP16 Range Validation and Clipping (opensearch-project#1562) * Add Range Validation for SQFP16 (opensearch-project#1493) * Add Range Validation for SQFP16 Vector Data Signed-off-by: Naveen Tatikonda <[email protected]> * Add index setting to clip vector data to FP16 range Signed-off-by: Naveen Tatikonda <[email protected]> * Add CHANGELOG Signed-off-by: Naveen Tatikonda <[email protected]> * Add an encoder parameter to clip fp16 range Signed-off-by: Naveen Tatikonda <[email protected]> * Address Review Comments Signed-off-by: Naveen Tatikonda <[email protected]> * Add BWC Tests Signed-off-by: Naveen Tatikonda <[email protected]> --------- Signed-off-by: Naveen Tatikonda <[email protected]> * SQFP16 Range Validation for Faiss IVF Models (opensearch-project#1557) * SQFP16 Range Validation for Faiss IVF Models Signed-off-by: Naveen Tatikonda <[email protected]> * Address Review Comments Signed-off-by: Naveen Tatikonda <[email protected]> --------- Signed-off-by: Naveen Tatikonda <[email protected]> * Rebase Changes Signed-off-by: Naveen Tatikonda <[email protected]> --------- Signed-off-by: Naveen Tatikonda <[email protected]> (cherry picked from commit d63ce27) * Fix EntityUtils package Signed-off-by: Naveen Tatikonda <[email protected]> --------- Signed-off-by: Naveen Tatikonda <[email protected]> Co-authored-by: Naveen Tatikonda <[email protected]>
…earch-project#1565) * Add release notes for 2.13.0 release Signed-off-by: Varun Jain <[email protected]> * Add release notes for 2.13.0 release Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]> (cherry picked from commit 9a38cfc) Co-authored-by: Varun Jain <[email protected]>
…earch-project#1568) Signed-off-by: Varun Jain <[email protected]> (cherry picked from commit e8c9ced) Co-authored-by: Varun Jain <[email protected]>
…pensearch-project#1579) * Add no new development notice to perf tool and OSB Added notice to inform developers to consider adding new features to OSB Add no feature development notice to OSB Signed-off-by: Vijayan Balasubramanian <[email protected]> (cherry picked from commit 17bbbd9) Co-authored-by: Vijayan Balasubramanian <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]>
…ctors to JNI layer functionality. (opensearch-project#1583) (opensearch-project#1584) Signed-off-by: Navneet Verma <[email protected]> (cherry picked from commit cc91c7c) Co-authored-by: Navneet Verma <[email protected]>
…project#1549) (opensearch-project#1580) Signed-off-by: panguixin <[email protected]> (cherry picked from commit c861966) Co-authored-by: panguixin <[email protected]>
Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit 653e7eb)
) (opensearch-project#1587) * support script score when doc value is disabled Signed-off-by: panguixin <[email protected]> * add test Signed-off-by: panguixin <[email protected]> * apply review comments Signed-off-by: panguixin <[email protected]> * fix test Signed-off-by: panguixin <[email protected]> --------- Signed-off-by: panguixin <[email protected]> (cherry picked from commit 771c4b5)
…layer to enable creation of larger segments for vector indices (opensearch-project#1604) (opensearch-project#1608) Changes include: 1. Add the interface for streaming the vectors from java to jni layer with initial capacity (opensearch-project#1586) 2. Integrating storeVectors interfaces with createIndex and createIndexTemplate functions. (opensearch-project#1588) 3. Update KNN80BinaryDocValues reader count live docs and use live docs as initial capacity to initialize vector address(opensearch-project#1595) 4. Move free vectorAddress from Java to JNI layer to reduce the memory footprint for Nmslib (opensearch-project#1602) Signed-off-by: Navneet Verma <[email protected]>
… minor optimization in KNNCodec (opensearch-project#1613) (opensearch-project#1615) Signed-off-by: Navneet Verma <[email protected]> (cherry picked from commit b48ab30) Co-authored-by: Navneet Verma <[email protected]>
* [k-NN] Add Clear Cache API (opensearch-project#740) * Add Clear Cache API Signed-off-by: Naveen Tatikonda <[email protected]> * Add Unit and Integration tests Signed-off-by: Naveen Tatikonda <[email protected]> * Add BWC Tests Signed-off-by: Naveen Tatikonda <[email protected]> * Add CHANGELOG Signed-off-by: Naveen Tatikonda <[email protected]> * Address Review Comments Signed-off-by: Naveen Tatikonda <[email protected]> --------- Signed-off-by: Naveen Tatikonda <[email protected]> (cherry picked from commit 12f4a51) Signed-off-by: Naveen Tatikonda <[email protected]> * Fix Failing Clear Cache API Test (opensearch-project#1336) Signed-off-by: Naveen Tatikonda <[email protected]> --------- Signed-off-by: Naveen Tatikonda <[email protected]> Co-authored-by: Naveen Tatikonda <[email protected]>
) Creates a new shared non-jni library that the jni libraries can depend on. This is to resolve a build issue on windows that is coming from nmslib not being able to find symbols in the jni common library. The net result will be that we add one more shared library. In addition, I removed all the CPack code from CMakeLists because we dont use it. Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit dc0953a)
…search-project#1626) * Add arm64 check when SIMD is disabled Signed-off-by: Ryan Bogan <[email protected]> * Add Changelog entry Signed-off-by: Ryan Bogan <[email protected]> * Change entry in changelog to infrastructure Signed-off-by: Ryan Bogan <[email protected]> --------- Signed-off-by: Ryan Bogan <[email protected]> (cherry picked from commit a2bb6cc) Co-authored-by: Ryan Bogan <[email protected]>
Signed-off-by: Junqiu Lei <[email protected]>
…ch-project#1637) (cherry picked from commit 730b7e4) Signed-off-by: Naveen Tatikonda <[email protected]> Co-authored-by: Naveen Tatikonda <[email protected]>
Fixes bug where we were not creating stored field type for knn_vector even when the mapping parameter is passed. Along with this, clean up the field mapper implementations. Add relevant uTs and iTs to ensure functionality is working as expected. (cherry picked from commit 699510d) Signed-off-by: John Mazanec <[email protected]>
…pensearch-project#1644) * Remove transport calls in TrainingJobRunner and TrainingJobClusterStateListener Signed-off-by: Ryan Bogan <[email protected]> * Fix tests Signed-off-by: Ryan Bogan <[email protected]> * Add changelog Signed-off-by: Ryan Bogan <[email protected]> * Fix CMake Faiss bug Signed-off-by: Ryan Bogan <[email protected]> * Add state checks for existing cluster metadata calls Signed-off-by: Ryan Bogan <[email protected]> * Remove CMake bug fix Signed-off-by: Ryan Bogan <[email protected]> * Fix changelog Signed-off-by: Ryan Bogan <[email protected]> * Fix failing tests Signed-off-by: Ryan Bogan <[email protected]> * Refactor and add two more created state checks Signed-off-by: Ryan Bogan <[email protected]> * Rebase and fix new tests Signed-off-by: Ryan Bogan <[email protected]> * Refactor created checks and modify error messages Signed-off-by: Ryan Bogan <[email protected]> * Refactor cluster state listener transport calls Signed-off-by: Ryan Bogan <[email protected]> --------- Signed-off-by: Ryan Bogan <[email protected]> (cherry picked from commit dc8eb6b) Co-authored-by: Ryan Bogan <[email protected]>
Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 11342be) Co-authored-by: Peter Zhu <[email protected]>
By commiting the patch changes, this commit prevents the libraries to be built from scratch every time cmake is ran. This will save additional build time. In addition to this, this commit separates out the different functionality to make the build system more readable. We can continue to iterate on this in the future. Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit c35f6ad) Co-authored-by: John Mazanec <[email protected]>
* Ensure ignore_unmapped is set to correct value Signed-off-by: Ryan Bogan <[email protected]> * Add changelog entry Signed-off-by: Ryan Bogan <[email protected]> --------- Signed-off-by: Ryan Bogan <[email protected]>
…1656) Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit b3829c3)
…#1657) - Adds clarifications for developing on Mac M series - Minor misc updates Signed-off-by: Tejas Shah <[email protected]> (cherry picked from commit f51dd64) Co-authored-by: Tejas Shah <[email protected]>
…search-project#1652) (opensearch-project#1658) * Use 0.95 as default ratio for lucene radial search traversal similarity (opensearch-project#1619) Signed-off-by: Junqiu Lei <[email protected]>
…roject#1593] (opensearch-project#1601) (opensearch-project#1663) Signed-off-by: Navneet Verma <[email protected]> (cherry picked from commit 5de10fc) Co-authored-by: Navneet Verma <[email protected]>
…ect#1661) Signed-off-by: Ryan Bogan <[email protected]>
…project#1662) (opensearch-project#1666) Signed-off-by: Ryan Bogan <[email protected]> (cherry picked from commit bd2f403) Co-authored-by: Ryan Bogan <[email protected]>
…arch-project#2452) Signed-off-by: Balasubramanian <[email protected]>
…nsearch-project#2461) Signed-off-by: AnnTian Shao <[email protected]> Co-authored-by: AnnTian Shao <[email protected]>
…#2449) Generates the vector source in the source field from the KnnVectorsFormat or BVD. It does this by adding StoredFieldsFormat to our existing custom codec. Currently, feature is experimental and behind a feature flag via index setting. In the future, we need to iterate to improve performance and stability for nested/object portions. Signed-off-by: John Mazanec <[email protected]>
* Adds debug logs for KNNQuery and KNNWeight Signed-off-by: Tejas Shah <[email protected]> * Adds check to see if log is enabled to start and stop StopWatch Signed-off-by: Tejas Shah <[email protected]> * Addressing comments on the PR Signed-off-by: Tejas Shah <[email protected]> * Adds shard and segment info in the logs Signed-off-by: Tejas Shah <[email protected]> * Removes unnecessary segment name param from exact search Signed-off-by: Tejas Shah <[email protected]> * Fixes the build Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Tejas Shah <[email protected]>
…rch-project#2468) Signed-off-by: Mark Wu <[email protected]> (cherry picked from commit d142366)
…ch-project#2474) Signed-off-by: John Mazanec <[email protected]> (cherry picked from commit abae323) Co-authored-by: John Mazanec <[email protected]>
…2477) Signed-off-by: Kunal Kotwani <[email protected]>
opensearch-project#2487) * Adding Vikasht34 as the maintainer in k-NN repo * Update MAINTAINERS.md --------- Signed-off-by: Navneet Verma <[email protected]> Signed-off-by: Peter Zhu <[email protected]> Co-authored-by: Peter Zhu <[email protected]>
… qty settings (opensearch-project#2480) Signed-off-by: Navneet Verma <[email protected]>
Signed-off-by: Kunal Kotwani <[email protected]>
Signed-off-by: Kunal Kotwani <[email protected]>
Signed-off-by: Kannan R <[email protected]>
…urce (opensearch-project#2497) Fixes a bug in the derived source writer where we are reading the entire bytes array from the bytes ref instead of just the offset+length. Also reuses readers to prevent memory leak Along with that, touches up the ParentChildHelper (no prod impact) and also adds some unit tests. Signed-off-by: John Mazanec <[email protected]>
…ith different gcc versions (opensearch-project#2506) (opensearch-project#2508) * Update build gradle to separate nmslib / faiss generation Signed-off-by: Peter Zhu <[email protected]> * Update scripts/build.sh to separate enable gcc switch Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Remove test comments Signed-off-by: Peter Zhu <[email protected]> * Updating restart and rolling upgrade bwc test bundle.gradle Signed-off-by: Peter Zhu <[email protected]> * Enforce gcc10 for nmslib to compile and avx512_spr have no-op Signed-off-by: Peter Zhu <[email protected]> --------- Signed-off-by: Peter Zhu <[email protected]> (cherry picked from commit 107c4f1) Co-authored-by: Peter Zhu <[email protected]>
Signed-off-by: Kunal Kotwani <[email protected]>
kotwanikunal
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
jmazanec15,
naveentatikonda,
junqiu-lei,
martin-gaievski,
ryanbogan,
luyuncheng,
shatejas,
0ctopus13prime and
Vikasht34
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Related Issues
N/A
Check List
New functionality includes testing.New functionality has been documented.API changes companion pull request created.--signoff.Public documentation issue/PR created.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.