Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add iterator support to pyknowhere interface. #189

Merged
merged 3 commits into from
Nov 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 43 additions & 1 deletion python/knowhere/knowhere.i
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,13 @@ import_array();
%include <std_pair.i>
%include <std_map.i>
%include <std_shared_ptr.i>
%include <std_vector.i>
%include <exception.i>
%shared_ptr(knowhere::DataSet)
%shared_ptr(knowhere::BinarySet)
%template(DataSetPtr) std::shared_ptr<knowhere::DataSet>;
%template(BinarySetPtr) std::shared_ptr<knowhere::BinarySet>;
%template(int64_float_pair) std::pair<long long int, float>;
%include <knowhere/expected.h>
%include <knowhere/dataset.h>
%include <knowhere/binaryset.h>
Expand Down Expand Up @@ -104,7 +106,7 @@ del Enum
%inline %{

class GILReleaser {
public:
public:
GILReleaser() : save(PyEval_SaveThread()) {
}
~GILReleaser() {
Expand All @@ -113,6 +115,28 @@ public:
PyThreadState* save;
};

class AnnIteratorWrap {
public:
AnnIteratorWrap(std::shared_ptr<IndexNode::iterator> it = nullptr) : it_(it) {
Copy link
Collaborator

@cydrain cydrain Nov 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest to assert here if it is NULL

if (it_ == nullptr) {
throw std::runtime_error("ann iterator must not be nullptr.");
}
}
~AnnIteratorWrap() {
}

bool HasNext() {
return it_->HasNext();
}

std::pair<int64_t, float> Next() {
return it_->Next();
}

private:
std::shared_ptr<IndexNode::iterator> it_;
};

class IndexWrap {
public:
IndexWrap(const std::string& name, const int32_t& version) {
Expand Down Expand Up @@ -157,6 +181,22 @@ class IndexWrap {
}
}

std::vector<AnnIteratorWrap>
GetAnnIterator(knowhere::DataSetPtr dataset, const std::string& json, const knowhere::BitsetView& bitset, knowhere::Status& status) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to "GetIterator"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping keyword Ann as discussed.

GILReleaser rel;
auto res = idx.AnnIterator(*dataset, knowhere::Json::parse(json), bitset);
std::vector<AnnIteratorWrap> result;
if (!res.has_value()) {
status = res.error();
return result;
}
status = knowhere::Status::success;
for (auto it : res.value()) {
result.emplace_back(it);
}
return result;
}

knowhere::DataSetPtr
RangeSearch(knowhere::DataSetPtr dataset, const std::string& json, const knowhere::BitsetView& bitset, knowhere::Status& status){
GILReleaser rel;
Expand Down Expand Up @@ -468,3 +508,5 @@ SetSimdType(const std::string type) {
}

%}

%template(AnnIteratorWrapVector) std::vector<AnnIteratorWrap>;
2 changes: 1 addition & 1 deletion python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ def get_readme():
description=(
"A library for efficient similarity search and clustering of vectors."
),
url="https://github.com/milvus-io/knowhere",
url="https://github.com/zilliztech/knowhere",
author="Milvus Team",
author_email="[email protected]",
license='Apache License 2.0',
Expand Down
1 change: 1 addition & 0 deletions src/common/config.cc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ static const std::unordered_set<std::string> ext_legal_json_keys = {"metric_type
"M", // HNSW param
"efConstruction", // HNSW param
"ef", // HNSW param
"seed_ef", // HNSW iterator param
"level",
"index_type",
"index_mode",
Expand Down
Loading