Implement model registry inference service reconciliation #135

lampajr · 2024-01-04T11:39:52Z

Fixes https://github.com/opendatahub-io/model-registry/issues/249

Description

Implement the model registry InferenceService reconciliation, the implemented workflow is the opposite of the one proposed in #124. Here the direction goes from Cluster to ModelRegistry.

The new reconciler will monitor InferenceService CRs having pre-defined labels, based on those labels will sync the model registry by keeping track of every deployment that occurred in the cluster.
Then will update the InferenceService CR by linking it to the model registry record using a specific label.

How Has This Been Tested?

e2e tests

Added e2e tests under test/e2e/ folder, currently I am running those tests using Kind cluster:

Setup cluster: kind create cluster --config test/config/kind-e2e-config.yaml
Create temporary dev tag image: IMG=quay.io/$USER/odh-model-controller:$(git rev-parse HEAD)
Build and tag dev image: make IMG=$IMG docker-build
Add local image to cluser: kind load docker-image $IMG
Deploy the controller: make IMG=${IMG} deploy-dev
Run e2e tests: make e2e-test

NOTE: These steps could be (hopefully) easily converted in a GHA if we decide to go with that.

manual test

Setup cluster

Install ODH 2.4.0
Install ODH component using the following DataScienceCluster CRD

kind: DataScienceCluster
apiVersion: datasciencecluster.opendatahub.io/v1
metadata:
  name: default
  labels:
    app.kubernetes.io/name: datasciencecluster
    app.kubernetes.io/instance: default
    app.kubernetes.io/part-of: opendatahub-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: opendatahub-operator
spec:
  components:
    codeflare:
      managementState: Removed
    dashboard:
      managementState: Managed
    datasciencepipelines:
      managementState: Removed
    kserve:
      managementState: Managed
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: overlays/odh
            uri: https://github.com/opendatahub-io/kserve/tarball/master
          - contextDir: config
            uri: https://github.com/lampajr/odh-model-controller/tarball/lampajr20231219_gh249_reconciler_from_isvc_test
    modelmeshserving:
      managementState: Managed
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: overlays/odh
            uri: https://github.com/opendatahub-io/modelmesh-serving/tarball/main
          - contextDir: config
            uri: https://github.com/lampajr/odh-model-controller/tarball/lampajr20231219_gh249_reconciler_from_isvc_test
    ray:
      managementState: Removed
    workbenches:
      managementState: Managed
    trustyai:
      managementState: Removed

NOTE: lampajr20231219_gh249_reconciler_from_isvc_test contains manifest changes to make use of the correct odh-model-controller image: alampare/odh-model-controller

By default this new model-registry for model-serving reconciler/controller is disabled in odh-model-controller, in order to properly enable it you should just start the controller with --model-registry-enabled flag.
https://github.com/lampajr/odh-model-controller/tarball/lampajr20231219_gh249_reconciler_from_isvc_test already contains this change in the configuration.

Install model registry operator
Clone model-registry-operator repository: git clone https://github.com/opendatahub-io/model-registry-operator.git
Install CRDs: make KUBECTL=oc install
Deploy the operator: make KUBECTL=oc deploy

Setup DS project

Create a Data Science project using the ODH dashboard, e.g., demo-model-registry-20240104
Setup a data connection (you could install a local minio instance in the newly create DS project)
Create a model server using the ODH dashboard
Apply the sample model registry CR from the model-registry-operator folder: kustomize build config/samples | oc apply -f - (make sure oc is pointing to the data science project create in step 1.)
Create a new model server using the ODH dashboard, in my case named model-server

Upload models

For the sake of simplicity let's upload some onnx files in the local minio instance (bucket name is models):

models/mnist/v12/mnist-12.onnx
models/mnist/v8/mnist-8.onnx

Fill up the model registry

NOTE: IDs could change depending on the order of operations and on the number of models you are going to register.

Set the model registry proxy url, you might have to setup a custom manual route from port :8080 of the model registry proxy

MR_HOSTNAME="<endpoint>"

Register an empty model, e.g, MNIST

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/registered_models" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "MNIST model for recognizing handwritten digits",
  "name": "MNIST",
  "state": "LIVE"
}'

Add a new model version

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "MNIST model version n. 8",
  "name": "v8",
  "registeredModelID": "1",
  "state": "LIVE"
}'

Add the model artifact to the created model version

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions/2/artifacts" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "Model artifact for MNIST v8",
  "uri": "s3://models/mnist/v8/mnist-8.onnx",
  "state": "UNKNOWN",
  "name": "v8-model",
  "modelFormatName": "onnx",
  "modelFormatVersion": "1",
  "storageKey": "aws-connection-models",
  "storagePath": "mnist/v8",
  "artifactType": "model-artifact"
}'

[Optional] If for some reason you need to update the artifact you can run:

curl --silent -X 'PATCH' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_artifacts/1" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '  {
    "description": "Model artifact for MNIST v8",
    "modelFormatName": "onnx",
    "modelFormatVersion": "1",
    "state": "UNKNOWN",
    "storageKey": "aws-connection-models",
    "storagePath": "mnist/v8",
    "uri": "s3://models/mnist/v8/mnist-8.onnx"
  }
'

NOTE: modelFormatName, modelFormatVersion, storageKey and storagePath are all those information needed to create the InferenceService so they should be valid values. E.g., the storageKey must match the existing data connection name (which is aws-connection-{{connection name}}).

You can use the following curls to inspect model registry content:

# Get registered models
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/registered_models?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

# Get model versions for a specific registered model
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

# Get model artifacts for a specific model version
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions/3/artifacts?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

# Get all inference services stored in model registry
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

Test model controller workflow

As soon as you create an InferenceService CR in the project, the odh-model-controller will create the ServingEnvironment in the model registry having the name == namespace (i.e., DS project), you can inspect its ID:

curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/serving_environments?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

Apply the InferenceService with proper links to the registered model (IDs might be different):

oc apply -n demo-model-registry-20240104 -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "mnist-v8-model"
  annotations:
    "openshift.io/display-name": "mnist-v8-model"
    "serving.kserve.io/deploymentMode": "ModelMesh"
  labels:
    "modelregistry.opendatahub.io/registered-model-id": "1"
    "modelregistry.opendatahub.io/model-version-id": "2"
    "opendatahub.io/dashboard": "true"
spec:
  predictor:
    model:
      modelFormat:
        name: onnx
        version: "1"
      runtime: model-server
      storage:
        key: aws-connection-models
        path: mnist/v8/mnist-8.onnx
EOF

modelregistry.opendatahub.io/registered-model-id identifies the registered model we would like to deploy
modelregistry.opendatahub.io/model-version-id identifies the specific version to deploy, if omitted (as in this case) let's deploy the latest version for that registered model

NOTE: the IDs must match, therefore ensure you are providing the corrects IDs.

At this point, the odh-model-controller will monitor InferenceService CRs and based on their labels will create the corresponding records in model registry

To check the model registry you can run:

curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

Expected output:

[
  {
    "createTimeSinceEpoch": "1704383784228",
    "customProperties": {},
    "id": "4",
    "lastUpdateTimeSinceEpoch": "1704383784228",
    "modelVersionId": "3",
    "name": "mnist-v8-model/4a2b96b4-af76-4270-bd2d-1a5aed2cd6ad",
    "registeredModelId": "2",
    "runtime": "model-server",
    "servingEnvironmentId": "1",
    "desiredState": "DEPLOYED"
  }
]

Check the ISVC has been correctly linked to the newly created record in model registry:

oc get inferenceservices.serving.kserve.io/mnist-v8-model -o yaml

Expected output (note modelregistry.opendatahub.io/inference-service-id label):

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  finalizers:
  - modelregistry.opendatahub.io/finalizer
  generation: 1
  labels:
    modelregistry.opendatahub.io/inference-service-id: 4
    modelregistry.opendatahub.io/registered-model-id": 1
    modelregistry.opendatahub.io/model-version-id": 2
    opendatahub.io/dashboard: true
  name: mnist-v8-model
  namespace: demo-model-registry-20240104
spec:
  predictor:
    model:
      modelFormat:
        name: onnx
        version: "1"
      name: ""
      resources: {}
      runtime: model-server
      storage:
        key: aws-connection-models
        path: mnist/v8

Delete the InferenceService (you can do this using the ODH dashboard as well)

oc delete -n demo-model-registry-20240104 -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "mnist-v8-model"
  annotations:
    "openshift.io/display-name": "mnist-v8-model"
    "serving.kserve.io/deploymentMode": "ModelMesh"
  labels:
    "modelregistry.opendatahub.io/registered-model-id": "1"
    "modelregistry.opendatahub.io/model-version-id": "2"
    "opendatahub.io/dashboard": "true"
spec:
  predictor:
    model:
      modelFormat:
        name: onnx
        version: 1
      runtime: model-server
      storage:
        key: aws-connection-models
        path: mnist/v8
EOF

Check record in model registry is correctly update, note the UNDEPLOYED state:

curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

Expected output:

[
  {
    "createTimeSinceEpoch": "1704383784228",
    "customProperties": {},
    "desiredState": "UNDEPLOYED",
    "id": "4",
    "lastUpdateTimeSinceEpoch": "1704384020813",
    "modelVersionId": "3",
    "name": "mnist-v8-model/4a2b96b4-af76-4270-bd2d-1a5aed2cd6ad",
    "registeredModelId": "2",
    "runtime": "model-server",
    "servingEnvironmentId": "1"
  }
]

NOTE step Upload models and Fill up the model registry could be automated using either notebooks or pipelines, please follow a more complete e2e demo at https://github.com/tarilabs/demo20231121#demo

Merge criteria:

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

openshift-ci · 2024-01-04T11:39:57Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

lampajr · 2024-01-12T12:58:51Z

As agreed we decided to implement this workflow where the model-controller has a more "passive" behavior with respect to the model registry reconcilication.

As described in the PR description, the new reconciler will monitor the InferenceService CRs that have proper labels defined and based on those values it will sync the deployments to model registry in the form of model registry InferenceService objects.

NOTE: under test/ folder I setup some e2e tests that I was used to run locally (added step-by-step instructions in the description) - if you think it could make sense I can add a github workflow to automatically run them using Kind

PS: all commits must be squashed as soon as the PR is approved or directly during the merge

lampajr · 2024-01-12T14:54:57Z

fyi @tarilabs @dhirajsb @rareddy

Jooho · 2024-01-15T14:41:06Z

test/data/deploy/inference-service-to-delete.yaml

+  namespace: default
+  labels:
+    "mr-inference-service-id": "4"
+  finalizers:


This finalizer will be added to the isvc created by mr controller? The finalizer of the isvc will not be removed by mr controller, right?

mmm not exactly, the finalizer will be added my the odh-model-controller itself (see https://github.com/opendatahub-io/odh-model-controller/pull/135/files#diff-e628c4483237f3dba685d802623c535845221a01b2bd2e3782a82ab9df48320aR120-R129) and then it will be removed by the model controller as well during the deletion.

The ISVC are going to be created by the dashboard as it is right now

test/data/deploy/inference-service-with-model-version.yaml

test/data/deploy/inference-service-without-model-version.yaml

Signed-off-by: Andrea Lamparelli <[email protected]>

In the scope of testing of Model Registry in openshift-ci: - make a Shell script which invokes some REST calls to MR, - so to make sure the REST endpoint is responsive, - then create a K8s ISVC on the cluster, - and display the MR InferenceService entities. Later, in a subsequent issue/PR, once: opendatahub-io/odh-model-controller#135 is merged, the last bulletpoint can be automated and placed under test in the final part of this script so to make sure the K8s ISVC on the cluster reflected as a precise MR InferenceService entity.

Co-authored-by: Edgar Hernández <[email protected]> Signed-off-by: Andrea Lamparelli <[email protected]>

Signed-off-by: Andrea Lamparelli <[email protected]>

config/overlays/e2e/kustomization.yaml

israel-hdez · 2024-02-09T00:32:16Z

/retest

israel-hdez

@lampajr I'm good with the changes.

I haven't tried this live. I'll try it on my Friday.

I think the only pending comment is the one from Vedant about the manifests. Let me know when you are ready. If I don't see any issue in my live testing, I'll merge once you confirm this is ready to go.

In the scope of testing of Model Registry in openshift-ci: - make a Shell script which invokes some REST calls to MR, - so to make sure the REST endpoint is responsive, - then create a K8s ISVC on the cluster, - and display the MR InferenceService entities. Later, in a subsequent issue/PR, once: opendatahub-io/odh-model-controller#135 is merged, the last bulletpoint can be automated and placed under test in the final part of this script so to make sure the K8s ISVC on the cluster reflected as a precise MR InferenceService entity.

Signed-off-by: Andrea Lamparelli <[email protected]>

lampajr

I haven't tried this live. I'll try it on my Friday.

I think the only pending comment is the one from Vedant about the manifests. Let me know when you are ready. If I don't see any issue in my live testing, I'll merge once you confirm this is ready to go.

Thsnk @israel-hdez I hope I added all details with regard to "how to test" this feature in a real cluster.

@VedantMahabaleshwarkar I agree with all your comments and I think I was able to simplify the dev manifests with my latest commit but unfortunately I wasn't able to fix the crd/external issue - I mean I couldn't find a way to include all those CRDs just in the overlays/dev because of

Error: accumulating resources: 2 errors occurred:
	* accumulateFile error: "accumulating resources from '../../crd/external/serving.kserve.io_inferenceservices.yaml': security; file '/home/alampare/repos/odh-model-controller/config/crd/external/serving.kserve.io_inferenceservices.yaml' is not in or below '/home/alampare/repos/odh-model-controller/config/overlays/dev'"
	* loader.New error: "error loading ../../crd/external/serving.kserve.io_inferenceservices.yaml with git: url lacks host: ../../crd/external/serving.kserve.io_inferenceservices.yaml, dir: got file 'serving.kserve.io_inferenceservices.yaml', but '/home/alampare/repos/odh-model-controller/config/crd/external/serving.kserve.io_inferenceservices.yaml' must be a directory to be a root, get: invalid source string: ../../crd/external/serving.kserve.io_inferenceservices.yaml"

on the other hand I do not see where those crd/external/kustomization is used, hence I don't see why keeping all those CRDs enable would be an issue. Is it referenced from other repos?

lampajr · 2024-02-09T09:30:17Z

config/crd/external/kustomization.yaml

+- monitoring.coreos.com_servicemonitors.yaml
+- maistra.io_servicemeshmemberrolls.yaml
+- maistra.io_servicemeshmembers.yaml
+- telemetry.istio.io_telemetries.yaml


I would agree with you @VedantMahabaleshwarkar but I don't know how to reference to crd/external from the overlays/dev without directly changing crd/external/kustomization.yaml - I tried to directly add all CRDs into the overlays/dev/kustomization but I get:

Error: accumulating resources: 2 errors occurred: * accumulateFile error: "accumulating resources from '../../crd/external/serving.kserve.io_inferenceservices.yaml': security; file '/home/alampare/repos/odh-model-controller/config/crd/external/serving.kserve.io_inferenceservices.yaml' is not in or below '/home/alampare/repos/odh-model-controller/config/overlays/dev'" * loader.New error: "error loading ../../crd/external/serving.kserve.io_inferenceservices.yaml with git: url lacks host: ../../crd/external/serving.kserve.io_inferenceservices.yaml, dir: got file 'serving.kserve.io_inferenceservices.yaml', but '/home/alampare/repos/odh-model-controller/config/crd/external/serving.kserve.io_inferenceservices.yaml' must be a directory to be a root, get: invalid source string: ../../crd/external/serving.kserve.io_inferenceservices.yaml"

any idea?

* add Shell script openshift-ci make some REST call to MR In the scope of testing of Model Registry in openshift-ci: - make a Shell script which invokes some REST calls to MR, - so to make sure the REST endpoint is responsive, - then create a K8s ISVC on the cluster, - and display the MR InferenceService entities. Later, in a subsequent issue/PR, once: opendatahub-io/odh-model-controller#135 is merged, the last bulletpoint can be automated and placed under test in the final part of this script so to make sure the K8s ISVC on the cluster reflected as a precise MR InferenceService entity. * omit OCP_CLUSTER_NAME to be valorized once on openshift-ci

lampajr · 2024-02-14T09:40:29Z

Hi @israel-hdez @VedantMahabaleshwarkar , thanks a lot for all your suggestions!!

I think I was able to address all of them, I also tested again in a real cluster and it worked for me (following the instructions I provided in the PR description).

israel-hdez · 2024-02-15T05:22:02Z

/hold

israel-hdez

I was able to try this and runs fine.
This is low risk because it is turned off by default. So, I'll merge.

I left one comment for your consideration for later improvement.

israel-hdez · 2024-02-15T05:24:26Z

test/e2e/modelregistry_controller_e2e_test.go

+	}
+	Expect(grpcPort).ToNot(BeNil())
+
+	mlmdAddr = fmt.Sprintf("localhost:%d", *grpcPort)


I had to change the code because of this connection to localhost.
This is somewhat specific to Kind. I was able to workaround by doing a port-forward, but given the tests re-deploy the registry, the port-forward needs to be re-opened before each test.

Interesting, thanks for pointing this out! I will keep track of this in a separate issue 💪

openshift-ci · 2024-02-15T05:25:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: israel-hdez, lampajr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [israel-hdez]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label Jan 4, 2024

lampajr mentioned this pull request Jan 4, 2024

Setup model registry serving reconciliation #124

Closed

3 tasks

lampajr force-pushed the lampajr20231219_gh249_reconciler_from_isvc branch 6 times, most recently from 837a40e to 15935b0 Compare January 11, 2024 16:50

lampajr force-pushed the lampajr20231219_gh249_reconciler_from_isvc branch from 15935b0 to 5d1c108 Compare January 12, 2024 12:36

lampajr force-pushed the lampajr20231219_gh249_reconciler_from_isvc branch from 5d1c108 to 6c60f51 Compare January 12, 2024 14:54

lampajr marked this pull request as ready for review January 12, 2024 14:54

openshift-ci bot removed the do-not-merge/work-in-progress label Jan 12, 2024

openshift-ci bot requested review from israel-hdez and spolti January 12, 2024 14:54

Jooho reviewed Jan 15, 2024

View reviewed changes

test/data/deploy/inference-service-with-model-version.yaml Outdated Show resolved Hide resolved

Jooho reviewed Jan 15, 2024

View reviewed changes

test/data/deploy/inference-service-without-model-version.yaml Outdated Show resolved Hide resolved

openshift-merge-robot added the needs-rebase label Jan 26, 2024

lampajr added 9 commits January 26, 2024 15:25

Fix overlays/dev deployment

490c0c4

Signed-off-by: Andrea Lamparelli <[email protected]>

Setup InferenceService reconciliation to model registry

4c03910

Signed-off-by: Andrea Lamparelli <[email protected]>

Setup e2e test for new reconciler

6eba8e2

Signed-off-by: Andrea Lamparelli <[email protected]>

Enable new reconciler

d843346

Signed-off-by: Andrea Lamparelli <[email protected]>

Fix model registry inference service reconciler

3fe6458

Signed-off-by: Andrea Lamparelli <[email protected]>

Create serving env when not found

015f6c6

Signed-off-by: Andrea Lamparelli <[email protected]>

Retrieve MR namespace from ISVC label

609b1b5

Signed-off-by: Andrea Lamparelli <[email protected]>

Added Kind cluster configuration for local e2e tests

1a765ee

Signed-off-by: Andrea Lamparelli <[email protected]>

Fix MR InferenceService reconciler

e055b26

Signed-off-by: Andrea Lamparelli <[email protected]>

lampajr force-pushed the lampajr20231219_gh249_reconciler_from_isvc branch from 29f6c8c to e368740 Compare February 8, 2024 13:37

tarilabs mentioned this pull request Feb 8, 2024

Provide Shell script openshift-ci make some REST call to MR opendatahub-io/model-registry-bf4-kf#294

Closed

tarilabs mentioned this pull request Feb 8, 2024

add Shell script openshift-ci make some REST call to MR opendatahub-io/model-registry-bf4-kf#295

Merged

3 tasks

Apply suggestions from code review

c9cbe07

Co-authored-by: Edgar Hernández <[email protected]> Signed-off-by: Andrea Lamparelli <[email protected]>

lampajr force-pushed the lampajr20231219_gh249_reconciler_from_isvc branch from e368740 to c9cbe07 Compare February 8, 2024 14:07

lampajr and others added 3 commits February 8, 2024 15:28

Apply suggestions from code review

25db572

Co-authored-by: Edgar Hernández <[email protected]> Signed-off-by: Andrea Lamparelli <[email protected]>

Check if IS already exists in MR

9c4e7b4

Signed-off-by: Andrea Lamparelli <[email protected]>

Improve e2e testing

962bd76

Signed-off-by: Andrea Lamparelli <[email protected]>

VedantMahabaleshwarkar requested changes Feb 8, 2024

View reviewed changes

config/overlays/e2e/kustomization.yaml Outdated Show resolved Hide resolved

openshift-ci bot assigned VedantMahabaleshwarkar Feb 8, 2024

israel-hdez reviewed Feb 9, 2024

View reviewed changes

Refactor dev manifests

4cbfd11

Signed-off-by: Andrea Lamparelli <[email protected]>

lampajr commented Feb 9, 2024

View reviewed changes

lampajr requested review from israel-hdez and VedantMahabaleshwarkar February 14, 2024 09:40

openshift-ci bot added the do-not-merge/hold label Feb 15, 2024

israel-hdez approved these changes Feb 15, 2024

View reviewed changes

openshift-ci bot assigned israel-hdez Feb 15, 2024

openshift-ci bot added the lgtm label Feb 15, 2024

openshift-ci bot added the approved label Feb 15, 2024

israel-hdez merged commit 518cfa2 into opendatahub-io:main Feb 15, 2024
4 of 5 checks passed

lampajr deleted the lampajr20231219_gh249_reconciler_from_isvc branch February 15, 2024 13:38

tarilabs mentioned this pull request Mar 28, 2024

RHOAIENG-3943 Update Model Registry documentation opendatahub-io/architecture-decision-records#34

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement model registry inference service reconciliation #135

Implement model registry inference service reconciliation #135

lampajr commented Jan 4, 2024 •

edited

Loading

openshift-ci bot commented Jan 4, 2024

lampajr commented Jan 12, 2024 •

edited

Loading

lampajr commented Jan 12, 2024

Jooho Jan 15, 2024

lampajr Jan 15, 2024 •

edited

Loading

israel-hdez commented Feb 9, 2024

israel-hdez left a comment

lampajr left a comment

lampajr Feb 9, 2024

lampajr commented Feb 14, 2024

israel-hdez commented Feb 15, 2024

israel-hdez left a comment

israel-hdez Feb 15, 2024

lampajr Feb 15, 2024

openshift-ci bot commented Feb 15, 2024

Implement model registry inference service reconciliation #135

Implement model registry inference service reconciliation #135

Conversation

lampajr commented Jan 4, 2024 • edited Loading

Description

How Has This Been Tested?

e2e tests

manual test

Setup cluster

Setup DS project

Upload models

Fill up the model registry

Test model controller workflow

Merge criteria:

openshift-ci bot commented Jan 4, 2024

lampajr commented Jan 12, 2024 • edited Loading

lampajr commented Jan 12, 2024

Jooho Jan 15, 2024

Choose a reason for hiding this comment

lampajr Jan 15, 2024 • edited Loading

Choose a reason for hiding this comment

israel-hdez commented Feb 9, 2024

israel-hdez left a comment

Choose a reason for hiding this comment

lampajr left a comment

Choose a reason for hiding this comment

lampajr Feb 9, 2024

Choose a reason for hiding this comment

lampajr commented Feb 14, 2024

israel-hdez commented Feb 15, 2024

israel-hdez left a comment

Choose a reason for hiding this comment

israel-hdez Feb 15, 2024

Choose a reason for hiding this comment

lampajr Feb 15, 2024

Choose a reason for hiding this comment

openshift-ci bot commented Feb 15, 2024

lampajr commented Jan 4, 2024 •

edited

Loading

lampajr commented Jan 12, 2024 •

edited

Loading

lampajr Jan 15, 2024 •

edited

Loading