diff --git a/enhancements/sig-architecture/136-placement-cel-selector/README.md b/enhancements/sig-architecture/136-placement-cel-selector/README.md new file mode 100644 index 00000000..edf53dc9 --- /dev/null +++ b/enhancements/sig-architecture/136-placement-cel-selector/README.md @@ -0,0 +1,321 @@ +# Select clusters with CEL expressions + +## Release Signoff Checklist + +- [] Enhancement is `provisional` +- [] Design details are appropriately documented from clear requirements +- [] Test plan is defined +- [] Graduation criteria for dev preview, tech preview, GA +- [] User-facing documentation is created in [website](https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/) + +## Summary + +This proposal proposes an enhancement to the placement cluster selector, enabling users to select clusters with CEL expressions. + +## Motivation + +In the OCM placement predicate stage, the user can currently use a label selector or claim selector to selector clusters. For detailed usage see [Label/Claim selection](https://open-cluster-management.io/docs/concepts/placement/#predicates). It can meet most of the use cases. However, there are some more advanced needs recently that the current functionality cannot fulfill. For example, the user wants to filter clusters with label "version" > 1.30.0. It's not smart to list every supported version in the label selector. Another example is bin packing, the user may want to filter clusters with specific resource quantities before prioritizing, the resource quantity will come from another resource, eg `AddonPlacementScore` but not labels or claims. So we have this proposal to support selecting clusters with CEL expressions. + +### Goals + +- Enhance `Placement` API to allow users to select clusters with CEL expressions. + +### Non-Goals + +- Enhance the `AddonPlacementScore` to include the original resource value for selecting clusters that is not in the scope. + +## Proposal + +In this proposal, we will introduce the `celSelector` in the predicate section. It will be a list of `celExpressions`. The result of the expressions are ANDed. The result of `labelSelector`, `claimSelector` and `celSelector` is also ANDed. + +We will also introduce a CEL library, `managedCluster` is declared as a variable and can be used in CEL expressions. Custom functions like comparing versions and getting cluster resources will also be added. + +### User Stories + +1. The user can select clusters by fields other than labels and claims. For example, select clusters by field `managedCluster.Status.version.kubernetes`. +2. The user can use CEL [Standard macros](https://github.com/google/cel-spec/blob/master/doc/langdef.md#macros) and [Standard functions](https://github.com/google/cel-spec/blob/master/doc/langdef.md#standard-definitions) on the `managedCluster` fields. For example, select clusters with version 1.30.* and 1.31.* by regex `'^1\\.(30|31)\\.\\d+$'`. +3. The user can use CEL customized function to select clusters by version comparison. For example, select clusters whose version > 1.13.0. +4. The user can use CEL customized function to select clusters by `AddonPlacementScore`. For example, select clusters whose cpuAvailable < 2. + +## Design Details + +### API Changes + +Add the new field `celSelector` to the `clusterSelector`. `celSelector` will be list of `celExpressions` with string type. + +```go +type ClusterSelector struct { + // LabelSelector represents a selector of ManagedClusters by label + // +optional + LabelSelector metav1.LabelSelector `json:"labelSelector,omitempty"` + + // ClaimSelector represents a selector of ManagedClusters by clusterClaims in status + // +optional + ClaimSelector ClusterClaimSelector `json:"claimSelector,omitempty"` + + // CelSelector represents a selector of ManagedClusters by CEL expressions on ManagedCluster fields + // +optional + CelSelector ClusterCelSelector `json:"celSelector,omitempty"` +} + +// ClusterCelSelector is a list of CEL expressions. The expressions are ANDed. +type ClusterCelSelector struct { + // +optional + CelExpressions []string `json:"celExpressions,omitempty"` +} +``` + +### ManagedCluster CEL Library + +A ManagedClusterLib will be added. +- Variable `managedCluster` will be added. +- Function `scores` will be added. +- Function `versionIsGreaterThan`, `versionIsLessThan` will be added. +- Function `quantityIsGreaterThan`, `quantityIsLessThan` will be added. + +```go +// ManagedClusterLib defines the CEL library for ManagedCluster evaluation. +// It provides functions and variables for evaluating ManagedCluster properties +// and their associated resources. +// +// Variables: +// +// managedCluster +// +// Provides access to ManagedCluster properties. +// +// Functions: +// +// scores +// +// Returns a list of AddOnPlacementScoreItem for a given cluster and AddOnPlacementScore resource name. +// +// scores(, ) +// +// The returned list contains maps with the following structure: +// - name: string - The name of the score +// - value: int - The numeric score value +// - quantity: number|string - The quantity value, represented as: +// * number: for pure decimal values (e.g., 3) +// * string: for values with units or decimal places (e.g., "300Mi", "1.5Gi") +// +// Examples: +// +// managedCluster.scores("cpu-memory") // returns [{name: "cpu", value: 3, quantity: 3}, {name: "memory", value: 4, quantity: "300Mi"}] +// +// Version Comparisons: +// +// versionIsGreaterThan +// +// Returns true if the first version string is greater than the second version string. +// The version must follow Semantic Versioning specification (http://semver.org/). +// It can be with or without 'v' prefix (eg, "1.14.3" or "v1.14.3"). +// +// versionIsGreaterThan(, ) +// +// Examples: +// +// versionIsGreaterThan("1.25.0", "1.24.0") // returns true +// versionIsGreaterThan("1.24.0", "1.25.0") // returns false +// +// versionIsLessThan +// +// Returns true if the first version string is less than the second version string. +// The version must follow Semantic Versioning specification (http://semver.org/). +// It can be with or without 'v' prefix (eg, "1.14.3" or "v1.14.3"). +// +// versionIsLessThan(, ) +// +// Examples: +// +// versionIsLessThan("1.24.0", "1.25.0") // returns true +// versionIsLessThan("1.25.0", "1.24.0") // returns false +// +// Quantity Comparisons: +// +// quantityIsGreaterThan +// +// Returns true if the first quantity string is greater than the second quantity string. +// +// quantityIsGreaterThan(, ) +// +// Examples: +// +// quantityIsGreaterThan("2Gi", "1Gi") // returns true +// quantityIsGreaterThan("1Gi", "2Gi") // returns false +// quantityIsGreaterThan("1000Mi", "1Gi") // returns false +// +// quantityIsLessThan +// +// Returns true if the first quantity string is less than the second quantity string. +// +// quantityIsLessThan(, ) +// +// Examples: +// +// quantityIsLessThan("1Gi", "2Gi") // returns true +// quantityIsLessThan("2Gi", "1Gi") // returns false +// quantityIsLessThan("1000Mi", "1Gi") // returns true + +type ManagedClusterLib struct{} + +// CompileOptions implements cel.Library interface to provide compile-time options. +func (ManagedClusterLib) CompileOptions() []cel.EnvOption { + return []cel.EnvOption{ + // The input types may either be instances of `proto.Message` or `ref.Type`. + // Here we use func ConvertManagedCluster() to convert ManagedCluster to a Map. + cel.Variable("managedCluster", cel.MapType(cel.StringType, cel.DynType)), + + cel.Function("scores", + cel.MemberOverload( + "cluster_scores", + []*cel.Type{cel.DynType, cel.StringType}, + cel.ListType(cel.DynType), + cel.FunctionBinding(clusterScores)), + ), + + cel.Function("versionIsGreaterThan", + cel.MemberOverload( + "version_is_greater_than", + []*cel.Type{cel.StringType, cel.StringType}, + cel.BoolType, + cel.FunctionBinding(versionIsGreaterThan)), + ), + + cel.Function("versionIsLessThan", + cel.MemberOverload( + "version_is_less_than", + []*cel.Type{cel.StringType, cel.StringType}, + cel.BoolType, + cel.FunctionBinding(versionIsLessThan)), + ), + + cel.Function("quantityIsGreaterThan", + cel.MemberOverload( + "quantity_is_greater_than", + []*cel.Type{cel.StringType, cel.StringType}, + cel.BoolType, + cel.FunctionBinding(quantityIsGreaterThan)), + ), + + cel.Function("quantityIsLessThan", + cel.MemberOverload( + "quantity_is_less_than", + []*cel.Type{cel.StringType, cel.StringType}, + cel.BoolType, + cel.FunctionBinding(quantityIsLessThan)), + ), + } +} +``` + +This `ManagedClusterLib` will be added when `cel.NewEnv()`. + +```go +func NewEvaluator() (*Evaluator, error) { + env, err := cel.NewEnv( + cel.Lib(ManagedClusterLib{}), // Add the ManagedClusterLib to the CEL environment + ) + if err != nil { + return nil, fmt.Errorf("failed to create CEL environment: %w", err) + } + return &Evaluator{env: env}, nil +} +``` + +### Examples + +1. The user can select clusters by Kubernetes version listed in `managedCluster.Status.version.kubernetes`. + +```yaml +apiVersion: cluster.open-cluster-management.io/v1beta1 +kind: Placement +metadata: + name: placement1 + namespace: default +spec: + numberOfClusters: 3 + clusterSets: + - prod + predicates: + - requiredClusterSelector: + celSelector: + celExpressions: + - managedCluster.Status.version.kubernetes == "v1.30.0" +``` +2. The user can use CEL [Standard macros](https://github.com/google/cel-spec/blob/master/doc/langdef.md#macros) and [Standard functions](https://github.com/google/cel-spec/blob/master/doc/langdef.md#standard-definitions) on the `managedCluster` fields. For example, select clusters with version 1.30.* and 1.31.* by regex.. + +```yaml +apiVersion: cluster.open-cluster-management.io/v1beta1 +kind: Placement +metadata: + name: placement1 + namespace: default +spec: + numberOfClusters: 3 + clusterSets: + - prod + predicates: + - requiredClusterSelector: + celSelector: + celExpressions: + - managedCluster.metadata.labels["version"].matches('^1\\.(30|31)\\.\\d+$') +``` + +3. The user can use CEL customized functions `versionIsGreaterThan` and `versionIsLessThan` to select clusters by version comparison. For example, select clusters whose version > 1.13.0. + + +```yaml +apiVersion: cluster.open-cluster-management.io/v1beta1 +kind: Placement +metadata: + name: placement1 + namespace: default +spec: + numberOfClusters: 3 + clusterSets: + - prod + predicates: + - requiredClusterSelector: + celSelector: + celExpressions: + - managedCluster.metadata.labels["version"].versionIsGreaterThan("1.30.0") +``` + + +4. The user can use CEL customized function `score` to select clusters by `AddonPlacementScore`. For example, select clusters whose cpuAvailable score < 2. + + +```yaml +apiVersion: cluster.open-cluster-management.io/v1beta1 +kind: Placement +metadata: + name: placement1 + namespace: default +spec: + numberOfClusters: 3 + clusterSets: + - prod + predicates: + - requiredClusterSelector: + celSelector: + celExpressions: + - managedCluster.scores("default").filter(s, s.name == 'cpuAvailable').all(e, e.quantity > 4) + - managedCluster.scores("default").filter(s, s.name == 'memAvailable').all(e, e.quantity.quantityIsGreaterThan("100Mi")) +``` + + +### Test Plan + +- TBD + +### Graduation Criteria +N/A + +### Upgrade Strategy +N/A + +## Alternatives +N/A + + diff --git a/enhancements/sig-architecture/136-placement-cel-selector/metadata.yaml b/enhancements/sig-architecture/136-placement-cel-selector/metadata.yaml new file mode 100644 index 00000000..94790bbb --- /dev/null +++ b/enhancements/sig-architecture/136-placement-cel-selector/metadata.yaml @@ -0,0 +1,14 @@ +title: select-clusters-with-cel-expressions +authors: + - "@haoqing0110" +reviewers: + - "@qiujian16" +approvers: + - "@qiujian16" +creation-date: 2024-12-18 +last-updated: 2024-12-18 +status: provisional +see-also: + - "/enhancements/sig-architecture/6-placements" + - "/enhancements/sig-architecture/15-resourcebasedscheduling" + - "/enhancements/sig-architecture/32-extensiblescheduling" diff --git a/enhancements/sig-architecture/32-extensiblescheduling/32-extensiblescheduling.md b/enhancements/sig-architecture/32-extensiblescheduling/32-extensiblescheduling.md index 0f415ec0..c890642b 100644 --- a/enhancements/sig-architecture/32-extensiblescheduling/32-extensiblescheduling.md +++ b/enhancements/sig-architecture/32-extensiblescheduling/32-extensiblescheduling.md @@ -16,10 +16,11 @@ The proposed work provides an API to represent the score of the managed cluster, When implementing placement resource based scheduling, we find in some cases, the prioritizer needs extra data (more than the default value provided by `ManagedCluster`) to calculate the score of the managed cluster. For example, there is a requirement to schedule based on resource monitoring data from the cluster. -So we want a more extensible way to support scheduling based on customized scores. +So we want a more extensible way to support scheduling based on customized resource scores and value. ### Goals -- Design a new API(CRD) to contain the customized scores for each managed cluster. +- Design a new API(CRD) to contain the customized resource scores and value for each managed cluster. +- Let placement predicate support filtering clusters with the resource original value provided by the new API(CRD). - Let placement prioritizer support rating clusters with the customized scores provided by the new API(CRD). ### Non-Goals @@ -33,11 +34,11 @@ So we want a more extensible way to support scheduling based on customized score ### User Stories #### Story 1: Users could use the data pushed from each managed cluster to select clusters. - - On each managed cluster, there is a customized agent monitoring the resource usage (eg. CPU ratio) of the cluster. It will calculate a score and push the result to the hub. + - On each managed cluster, there is a customized agent monitoring the resource usage (eg. CPU ratio) of the cluster. It will calculate a score and push the result to the hub. The agent can also push the original value of the resource to the hub. - As an end user, I can configure placement yaml to use this score to select clusters. #### Story 2: Users could use the metrics collected on the hub to select clusters. - - On the hub, there is a customized agent to get metrics (eg. cluster allocatable memory) from Thanos. It will generate a score for each cluster. + - On the hub, there is a customized agent to get metrics (eg. cluster allocatable memory) from Thanos. It will generate a score for each cluster. The agent can also push the original value of the resource to the hub. - As an end user, I can configure placement yaml to use this score to select clusters. #### Story 3: Disaster recovery workload could be automatically switched to an available cluster. @@ -110,6 +111,10 @@ type AddOnPlacementScoreItem struct { // +kubebuilder:validation:Maximum:=100 // +required Value int32 `json:"value"` + + // Quantity defines the original value of the score. + // +optional + Quantity resource.Quantity `json:"quantity"` } // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object @@ -307,6 +312,9 @@ The scores inside `AddOnPlacementScore` are supposed to be updated frequently. R For the invalid score cases, for example, the AddOnPlacementScore CR is not created for some managed clusters, the score is missing in some CR, or the score is expired. The prioritizer will give those clusters with score 0, and the final placement decision is still made by the total score of each prioritizers. +### Let placement support filtering clusters with the original value of the CR. +Refer to [select clusters with CEL expressions](/enhancements/sig-architecture/136-placement-cel-selector/README.md). + ### How to maintain the lifecycle of the AddOnPlacementScore CRs? The details of how to maintain the AddOnPlacementScore CRs lifecycle is out of scope, however, this proposal would like to give some suggestions about how to implement a 3rd party controller for it. - Where should the 3rd party controller run? diff --git a/enhancements/sig-architecture/32-extensiblescheduling/metadata.yaml b/enhancements/sig-architecture/32-extensiblescheduling/metadata.yaml index 55d9d4c3..6254d2a4 100644 --- a/enhancements/sig-architecture/32-extensiblescheduling/metadata.yaml +++ b/enhancements/sig-architecture/32-extensiblescheduling/metadata.yaml @@ -10,8 +10,9 @@ approvers: - "@qiujian16" - "@elgnay" creation-date: 2021-11-01 -last-updated: 2022-01-28 +last-updated: 2025-01-08 status: implemented see-also: - "/enhancements/sig-architecture/6-placements" - "/enhancements/sig-architecture/15-resourcebasedscheduling" + - "/enhancements/sig-architecture/136-placement-cel-selector" \ No newline at end of file