Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Implementing online_read for MilvusOnlineStore #4996

Merged
merged 6 commits into from
Feb 1, 2025
Merged

feat: Implementing online_read for MilvusOnlineStore #4996

merged 6 commits into from
Feb 1, 2025

Conversation

franciscojavierarceo
Copy link
Member

@franciscojavierarceo franciscojavierarceo commented Jan 31, 2025 •
edited
Loading

What this PR does / why we need it:

This pull request includes several updates and improvements to the Feast project, focusing on the integration and enhancement of the Milvus online store. Below is a detailed summary of the changes made:

  1. Examples Update

    • README.md: Modified the materialization steps for features into the online store by switching to Python code snippets.
    • milvus-quickstart.ipynb: Format change.

  2. Milvus Online Store Enhancements

    • milvus.py:
      • Removed unused imports.
      • Added VALUE_TYPE_TO_PROTO_VALUE_MAP for mapping value types.
      • Introduced a new method _get_composite_key_name to generate composite key names.
      • Updated methods to utilize _get_composite_key_name for consistency.
      • Implemented the online_read method to handle reading from the Milvus online store.
      • Improved the _extract_proto_values_to_dict method to handle more numeric types and serialization.

  3. Type Map Enhancements

    • type_map.py:
      • Added VALUE_TYPE_TO_PROTO_VALUE_MAP for better type conversion.

  4. Feast Types Enhancements

    • types.py:
      • Introduced from_feast_type method to convert Feast types to ValueType.

  5. Example Repository Updates

    • example_feature_repo_1.py:
      • Updated the schema for document_embeddings to include vector index and search metric attributes.

  6. Unit Tests Enhancements

    • test_online_retrieval.py:
      • Modified test_get_online_features for better readability.
      • Added comprehensive tests for Milvus online features retrieval.
      • Added skipping marker for test_local_milvus due to CI issues.

Testing

  • Comprehensive unit tests have been added and modified to ensure the functionality of the changes made.
  • Manual testing was performed to validate the integration with the Milvus online store.

Which issue(s) this PR fixes:

Another for #4364

Misc

  • The test_local_milvus has been marked to be skipped due to CI struggles. Further investigation is needed to address this.

@franciscojavierarceo franciscojavierarceo marked this pull request as ready for review February 1, 2025 01:50
@franciscojavierarceo franciscojavierarceo requested a review from a team as a code owner February 1, 2025 01:50
lokeshrangineni
lokeshrangineni reviewed Feb 1, 2025
View reviewed changes
sdk/python/tests/unit/online_store/test_online_retrieval.py Outdated
full_feature_names=False,
)

# TODO: Need to fix these tests to actually run corecctly.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this needs to revisit now or later?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was going to handle that in a follow up PR too. Trying to get a release out soon was the main reason.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

@franciscojavierarceo
adding milvus md and incorporating feedback from Lokesh
edba589 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
including Milvus in summary page
8c605c1 
Signed-off-by: Francisco Javier Arceo <[email protected]>
@franciscojavierarceo
linter
cb71592 
Signed-off-by: Francisco Javier Arceo <[email protected]>
shuchu
shuchu approved these changes Feb 1, 2025
View reviewed changes
Copy link
Collaborator

@shuchu shuchu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@franciscojavierarceo franciscojavierarceo merged commit 92dde13 into master Feb 1, 2025
25 checks passed
@dmartinol
Copy link
Contributor

amazing example notebook, thank you! 💯

franciscojavierarceo pushed a commit that referenced this pull request Feb 4, 2025
@feast-ci-bot
chore(release): release 0.44.0
aa7224a 
# [0.44.0](v0.43.0...v0.44.0) (2025-02-04)

### Bug Fixes

* Adding periodic check to fix the sporadic failures of the operator e2e tests.  ([#4952](#4952)) ([1d086be](1d086be))
* Adding the feast-operator/bin to the .gitignore directory. Somehow it… ([#5005](#5005)) ([1a027ee](1a027ee))
* Changed Env Vars for e2e tests ([#4975](#4975)) ([fa0084f](fa0084f))
* Fix GitHub Actions to pass authentication ([#4963](#4963)) ([22b9138](22b9138)), closes [#4937](#4937) [#4939](#4939) [#4941](#4941) [#4940](#4940) [#4943](#4943) [#4944](#4944) [#4945](#4945) [#4946](#4946) [#4947](#4947) [#4948](#4948) [#4951](#4951) [#4954](#4954) [#4957](#4957) [#4958](#4958) [#4959](#4959) [#4960](#4960) [#4962](#4962)
* Fix showing selected navigation item in UI sidebar ([#4969](#4969)) ([8ac6a85](8ac6a85))
* Invalid column names in get_historical_features when there are field mappings on join keys ([#4886](#4886)) ([c9aca2d](c9aca2d))
* Read project data from the 'projects' key while loading the registry state in the Feast UI ([#4772](#4772)) ([cb81939](cb81939))
* Remove grpcurl dependency from Operator ([#4972](#4972)) ([439e0b9](439e0b9))
* Removed the dry-run flag to test and we will add it back later. ([#5007](#5007)) ([d112b52](d112b52))
* Render UI navigation items as links instead of buttons ([#4970](#4970)) ([1267703](1267703))
* Resolve Operator CRD bloat due to long field descriptions ([#4985](#4985)) ([7593bb3](7593bb3))
* Update manifest to add feature server image for odh ([#4973](#4973)) ([6a1c102](6a1c102))
* Updating release workflows to refer to yml instead of yaml ([#4935](#4935)) ([02b0a68](02b0a68))
* Use locally built feast-ui package in dev feature-server image ([#4998](#4998)) ([0145e55](0145e55))

### Features

* Added OWNERS file for OpenshiftCI ([#4991](#4991)) ([86a2ee8](86a2ee8))
* Adding Milvus demo to examples ([#4910](#4910)) ([2daf852](2daf852))
* Adding retrieve_online_documents endpoint ([#5002](#5002)) ([6607d3d](6607d3d))
* Adding support to return additional features from vector retrieval for Milvus db ([#4971](#4971)) ([6ce08d3](6ce08d3))
* Creating/updating the stable branch after the release. ([#5003](#5003)) ([e9b53cc](e9b53cc))
* Implementing online_read for MilvusOnlineStore ([#4996](#4996)) ([92dde13](92dde13))
* Improve exception message for unsupported Snowflake data types ([#4779](#4779)) ([5992364](5992364))
* Operator add feast ui deployment ([#4930](#4930)) ([b026d0c](b026d0c))
* Updating documents to highlight v2 api for Vector Similarity Se… ([#5000](#5000)) ([32b82a4](32b82a4))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lokeshrangineni lokeshrangineni lokeshrangineni left review comments

@shuchu shuchu shuchu approved these changes

Assignees
No one assigned
Labels
ok-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@franciscojavierarceo @dmartinol @shuchu @lokeshrangineni