✨ support config work driver. #256

morvencao · 2024-03-26T06:58:45Z

Summary

This PR aims to enhance the addon-manager by adding work driver configuration support.
This enables the addon manager to deploy addon agents using drivers other than kube.

Related issue(s)

ref: https://issues.redhat.com/browse/ACM-10423

morvencao · 2024-03-26T13:28:58Z

/hold

morvencao · 2024-03-27T13:40:26Z

/assign @qiujian16 @skeeey

qiujian16 · 2024-03-27T14:48:43Z

go.mod

 	open-cluster-management.io/api v0.13.0
-	open-cluster-management.io/sdk-go v0.13.0
+	open-cluster-management.io/ocm v0.13.1-0.20240325044258-22501d88f774


we should avoid import ocm

removed ocm dependency.

qiujian16

can we enable the integration for cloudevents？

qiujian16 · 2024-03-28T09:18:24Z

pkg/addonmanager/manager.go

+}
+
+// New returns a new addon manager for creating addon agents.
+func New(config *rest.Config, opts *ManagerOptions) (AddonManager, error) {


this will break the existing signature of the function. the opts filed should be an optional field. And we should allow that it is not set.

updated this to make managerOptions be optional.

qiujian16 · 2024-03-28T09:20:29Z

cmd/example/helloworld/main.go

@@ -66,22 +66,30 @@ func newCommand() *cobra.Command {
 }

 func newControllerCommand() *cobra.Command {
+	o := addonmanager.NewManagerOptions()


we should not impact the existing addon. We could build a new addon example that supports cloudevent, or select one addon as an example. It also ensure that existing addon won't be impacted by this change.

built new example that supports cloudevent, following the existing examples.

morvencao · 2024-04-15T06:19:16Z

test/integration-test.mk

+test-kube-integration: ensure-kubebuilder-tools
+	go test -c ./test/integration/kube -o ./kube-integration.test
+	./kube-integration.test -ginkgo.slowSpecThreshold=15 -ginkgo.v -ginkgo.failFast
+.PHONY: test-kube-integration
+
+test-cloudevents-integration: ensure-kubebuilder-tools
+	go test -c ./test/integration/cloudevents -o ./cloudevents-integration.test
+	./cloudevents-integration.test -ginkgo.slowSpecThreshold=15 -ginkgo.v -ginkgo.failFast
+.PHONY: test-cloudevents-integration
+
+test-integration: test-kube-integration test-cloudevents-integration


created separate folder for cloudevents integration test.
make sense? @qiujian16 @skeeey

yeah, we may do some configuration like this https://github.com/open-cluster-management-io/ocm/blob/main/.github/workflows/cloudevents-integration.yml#L5

qiujian16 · 2024-04-16T03:27:18Z

pkg/addonmanager/controllers/agentdeploy/utils.go

@@ -90,6 +92,9 @@ func newManifestWork(addonNamespace, addonName, clusterName string, manifests []
 			Labels: map[string]string{
 				addonapiv1alpha1.AddonLabelKey: addonName,
 			},
+			Annotations: map[string]string{
+				common.CloudEventsDataTypeAnnotationKey: payload.ManifestBundleEventDataType.String(),


We need to add this annotation all the time? I do not think we should add any additionaly annotation to existing manifestwork.

yes, we do need this all the time. /cc @skeeey

we need this annotation to indicate the manifest data type, but I think we may be able to enhance the sdk-go, if we don't find this annotation, we use ManifestBundle as a default value?

and for ocm/acm cases, it should be always ManifestBundle

yes, we can set default value(bundle type) in sdk-go so that we don't need set it here explicitly.

or we can give a fixed value(bundle type) now

updated sdk-go dependency, removed this annotation.

qiujian16 · 2024-04-16T03:29:38Z

pkg/addonmanager/manager.go

-// New returns a new Manager for creating addon agents.
-func New(config *rest.Config) (AddonManager, error) {
-	return &addonManager{
+// ManagerOptions defines the flags for addon manager


can we put this Option flag in the separated dir specifically for cloudevent?

put this to separate cloudevents package.

qiujian16 · 2024-04-16T03:34:03Z

pkg/addonmanager/manager.go

+	// ManifestWork client that implements the ManifestWorkInterface and ManifestWork informer based on different
+	// driver configuration.
+	// Refer to Event Based Manifestwork proposal in enhancements repo to get more details.
+	_, config, err := cloudeventswork.NewConfigLoader(a.options.WorkDriver, a.options.WorkDriverConfig).


could we move all this cloudEvent setting to a separated func instead of the main Start func? or a separated cloudEvent manager? I am concerned about this change on the existing addon-framework breaking existing addons.

Implemented cloudeventsAddonManager in separate package cloudevents, so that we don't break existing addons. see: dcf9539

Signed-off-by: morvencao <[email protected]>

morvencao · 2024-04-17T02:22:12Z

kindly ping @zhujian7 @zhiweiyin318 ~

zhiweiyin318 · 2024-04-17T08:23:54Z

Look good to me.

qiujian16 · 2024-04-17T08:36:58Z

pkg/addonmanager/cloudevents/manager.go

+	return fmt.Errorf("method StartWithInformers is not implemented")
+}
+
+func (a *cloudeventsAddonManager) startWithInformers(ctx context.Context,


actully, how about we have a baseManager interface with StartWithInformers method and an implementation that new all controllers. The Manager interface inlines the baseManager interface, and the manager/cloudeventmanger have different implementation of start func, but share the same implementation in baseManager. Will that work?

Updated this, not sure if this is more clear.

why we still need this? I thought this should be in the baseManager.

removed this, PTAL @qiujian16

I mean all the code to init controllers and start them. I think they are duplicated code and should exist only in baseManager. If we have any change on controllers in the future, we do not need to update multiple times.

We may not do that, due to the limitation source work client from sdk-go as I mentioned at #256 (comment)

we can't get the workinformers.SharedInformerFactory from clientHolder (from sdk-go), we can only get workv1informers.Informer type, so we can't call StartWithInformers to start the controllers in cloudevents addon manager.

Signed-off-by: morvencao <[email protected]>

zhujian7 · 2024-04-18T03:25:59Z

examples/deploy/ocm-cloudevents/install.sh

+      agentBroadcast: clusters/${MANAGED_CLUSTER_NAME}/agentbroadcast
+EOF
+
+# patch klusterlet-work-agent deployment to use mqtt as workload source driver


Can the klusterlet-work-agent be deployed in the way that the mqtt is the driver directly?

Yes, the latest work agent image already support mqtt and grpc drivers.

zhujian7 · 2024-04-18T03:31:56Z

examples/helloworld_cloudevents/helloworld.go

+	return addonfactory.StructToValues(manifestConfig), nil
+}
+
+func AgentHealthProber() *agent.HealthProber {


is there any specific reason that we need to use the work prober to check the health for cloud events driver addons?

the cloudevents driver added in this PR is only used for addon agent deploy, no other difference from the current "kube" driver. Therefore, it use similar prober to make sure addon agent is deployed successfully on managed cluster as "kube" driver. Make sense?

zhujian7 · 2024-04-18T04:33:47Z

pkg/addonmanager/manager.go

-	}
-	return nil
-}
-
 // New returns a new Manager for creating addon agents.
 func New(config *rest.Config) (AddonManager, error) {


Not sure if it is better to provider a addon manager factory func, like:

func New(mtype AddonManagerType)(AddonManager, error){ if mtype =="cloudevents"{ return ... } ... }

this will break existing code that using addonmanager.New(Config) to create addon manager, right?
we don't want make any change to current code, that's why we keep this unchanged.

zhujian7 · 2024-04-18T04:44:11Z

pkg/addonmanager/cloudevents/manager.go

+
+// cloudeventsAddonManager is the implementation of AddonManager with
+// the base implementation and cloudevents options
+type cloudeventsAddonManager struct {


could you explain the difference between the cloudevents addon manager and the classic addon manager except for the building way of manifestwork client and informer? which I think is good for review.

the difference between the cloudevents addon manager and the classic addon manager is the way to build work client and work informer.

the cloudevents addon manager use cloudevents drivers(mqtt or grpc) to create work informer, so that the manifestwork used to deploy addon agent can be delivered to managed cluster through mqtt or grpc.

no other difference.

zhujian7 · 2024-04-18T04:54:30Z

.github/workflows/cloudevents-integration.yml

+  contents: read
+
+jobs:
+  integration:


can we also add an e2e to at least check that the example cloud event addons are available when deploying into a kubernetes cluster?

e2e test was removed in this commit: d8d4f52
because the work client based on cloudevents drivers (sdk-go) are missing GC machnism, we will revisit and add e2e after GC machnism support for cloudevents drivers.

Signed-off-by: morvencao <[email protected]>

morvencao · 2024-04-18T09:50:20Z

@qiujian16 @zhujian7 Another look?

qiujian16

/approve

qiujian16 · 2024-04-19T02:06:26Z

pkg/addonmanager/cloudevents/manager.go

+	workInformers := clientHolder.ManifestWorkInformer()
+
+	// addonDeployController
+	err = workInformers.Informer().AddIndexers(


nit: the indexer part should be put in a shared dir. It is duplicated with the manager.

openshift-ci · 2024-04-19T02:08:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: morvencao, qiujian16

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [qiujian16]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

morvencao · 2024-04-19T05:41:21Z

added back the e2e testing for helloworld addon using cloudevents.
/unhold

@zhujian7 PTAL

Signed-off-by: morvencao <[email protected]>

zhujian7 · 2024-04-19T06:56:11Z

LGTM

zhujian7 · 2024-04-19T06:58:51Z

/lgtm

openshift-ci bot requested review from deads2k and zhiweiyin318 March 26, 2024 06:58

morvencao force-pushed the br_add_work_driver branch from 81f461b to cd70817 Compare March 26, 2024 07:43

openshift-ci bot added the do-not-merge/hold label Mar 26, 2024

openshift-merge-robot added the needs-rebase label Mar 27, 2024

morvencao force-pushed the br_add_work_driver branch from 5ab10d3 to ddf31a0 Compare March 27, 2024 12:37

openshift-merge-robot removed the needs-rebase label Mar 27, 2024

openshift-ci bot assigned qiujian16 and skeeey Mar 27, 2024

qiujian16 reviewed Mar 27, 2024

View reviewed changes

morvencao force-pushed the br_add_work_driver branch 3 times, most recently from 6cafea7 to 0007165 Compare March 28, 2024 08:06

qiujian16 reviewed Mar 28, 2024

View reviewed changes

morvencao force-pushed the br_add_work_driver branch 4 times, most recently from f7a589a to 4f7535b Compare April 15, 2024 04:04

morvencao commented Apr 15, 2024

View reviewed changes

morvencao force-pushed the br_add_work_driver branch 7 times, most recently from b480f3b to a5ebffe Compare April 16, 2024 02:48

qiujian16 reviewed Apr 16, 2024

View reviewed changes

morvencao force-pushed the br_add_work_driver branch from c5e39cb to dcf9539 Compare April 16, 2024 04:27

support cloudevents driver for addon manager.

da0d4f9

Signed-off-by: morvencao <[email protected]>

morvencao force-pushed the br_add_work_driver branch from dcf9539 to da0d4f9 Compare April 16, 2024 07:36

morvencao added 2 commits April 16, 2024 07:43

remove e2e testing for cloudevents.

d8d4f52

Signed-off-by: morvencao <[email protected]>

revert addon manager interface change.

5c1a6e7

Signed-off-by: morvencao <[email protected]>

qiujian16 reviewed Apr 17, 2024

View reviewed changes

morvencao force-pushed the br_add_work_driver branch from a81869c to 2b67a4d Compare April 17, 2024 11:08

add base addon manager.

7ad6107

Signed-off-by: morvencao <[email protected]>

morvencao force-pushed the br_add_work_driver branch from 2b67a4d to 7ad6107 Compare April 18, 2024 01:39

zhujian7 reviewed Apr 18, 2024

View reviewed changes

address comments.

a7bac3b

Signed-off-by: morvencao <[email protected]>

qiujian16 reviewed Apr 19, 2024

View reviewed changes

openshift-ci bot added the approved label Apr 19, 2024

openshift-ci bot removed the do-not-merge/hold label Apr 19, 2024

morvencao force-pushed the br_add_work_driver branch from 68ae426 to f61bf8b Compare April 19, 2024 05:46

add e2e test back.

c7ba5b9

Signed-off-by: morvencao <[email protected]>

morvencao force-pushed the br_add_work_driver branch from f61bf8b to c7ba5b9 Compare April 19, 2024 05:47

openshift-ci bot assigned zhujian7 Apr 19, 2024

openshift-ci bot added the lgtm label Apr 19, 2024

openshift-merge-bot bot merged commit e703fc5 into open-cluster-management-io:main Apr 19, 2024
14 checks passed

morvencao deleted the br_add_work_driver branch April 19, 2024 07:02

morvencao mentioned this pull request Apr 20, 2024

✨ update addon manager open-cluster-management-io/ocm#434

Merged

✨ support config work driver. #256

✨ support config work driver. #256

Conversation

morvencao commented Mar 26, 2024 • edited Loading

Summary

Related issue(s)

morvencao commented Mar 26, 2024

morvencao commented Mar 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiujian16 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morvencao commented Apr 17, 2024

zhiweiyin318 commented Apr 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morvencao Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morvencao commented Apr 18, 2024

qiujian16 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Apr 19, 2024

morvencao commented Apr 19, 2024 • edited Loading

zhujian7 commented Apr 19, 2024

zhujian7 commented Apr 19, 2024

morvencao commented Mar 26, 2024 •

edited

Loading

morvencao Apr 18, 2024 •

edited

Loading

morvencao commented Apr 19, 2024 •

edited

Loading