diff --git a/charts/katalyst/crds/config.katalyst.kubewharf.io_adminqosconfigurations.yaml b/charts/katalyst/crds/config.katalyst.kubewharf.io_adminqosconfigurations.yaml index 4035ca8..aba5787 100644 --- a/charts/katalyst/crds/config.katalyst.kubewharf.io_adminqosconfigurations.yaml +++ b/charts/katalyst/crds/config.katalyst.kubewharf.io_adminqosconfigurations.yaml @@ -24,6 +24,9 @@ spec: - jsonPath: .spec.nodeLabelSelector name: SELECTOR type: string + - jsonPath: .spec.priority + name: PRIORITY + type: string - jsonPath: .spec.ephemeralSelector.nodeNames name: NODES type: string @@ -63,10 +66,197 @@ spec: config: description: Config is custom field for admin qos configuration properties: + evictionConfig: + description: EvictionConfig is a configuration for eviction + properties: + cpuPressureEvictionConfig: + description: CPUPressureEvictionConfig is the config for cpu + pressure eviction + properties: + enableLoadEviction: + description: EnableLoadEviction is whether to enable cpu + load eviction + type: boolean + enableSuppressionEviction: + description: EnableSuppressionEviction is whether to enable + pod-level cpu suppression eviction + type: boolean + gracePeriod: + description: GracePeriod is the grace period of cpu pressure + eviction + format: int64 + type: integer + loadEvictionCoolDownTime: + description: LoadEvictionCoolDownTime is the cool-down + time of cpu load eviction, if the cpu load eviction + is triggered, the cpu load eviction will be disabled + for the cool-down time + type: string + loadMetricRingSize: + description: LoadMetricRingSize is the size of the load + metric ring, which is used to calculate the load of + the target cpuset pool + minimum: 1 + type: integer + loadThresholdMetPercentage: + description: LoadThresholdMetPercentage is the percentage + of the number of times the load over the upper bound + to the total number of times the load is measured, if + the percentage is greater than the load threshold met + percentage, the eviction or node tainted will be triggered + type: number + loadUpperBoundRatio: + description: LoadUpperBoundRatio is the upper bound ratio + of cpuset pool load, if the load of the target cpuset + pool is greater than the load upper bound repeatedly, + the eviction will be triggered + type: number + maxSuppressionToleranceRate: + description: MaxSuppressionToleranceRate is the maximum + cpu suppression tolerance rate that can be set by the + pod, if the cpu suppression tolerance rate of the pod + is greater than the maximum cpu suppression tolerance + rate, the cpu suppression tolerance rate of the pod + will be set to the maximum cpu suppression tolerance + rate + type: number + minSuppressionToleranceDuration: + description: MinSuppressionToleranceDuration is the minimum + duration a pod can tolerate cpu suppression, only if + the cpu suppression duration of the pod is greater than + the minimum cpu suppression duration, the eviction will + be triggered + type: string + type: object + dryRun: + description: DryRun is the list of eviction plugins to dryRun + '*' means "all dry-run by default" 'foo' means "dry-run + 'foo'" first item for a particular name wins + items: + type: string + type: array + memoryPressureEvictionConfig: + description: MemoryPressureEvictionConfig is the config for + memory pressure eviction + properties: + enableNumaLevelEviction: + description: EnableNumaLevelEviction is whether to enable + numa-level eviction + type: boolean + enableRSSOveruseEviction: + description: EnableRSSOveruseEviction is whether to enable + pod-level rss overuse eviction + type: boolean + enableSystemLevelEviction: + description: EnableSystemLevelEviction is whether to enable + system-level eviction + type: boolean + gracePeriod: + description: GracePeriod is the grace period of memory + pressure eviction + format: int64 + type: integer + numaEvictionRankingMetrics: + description: NumaEvictionRankingMetrics is the metrics + used to rank pods for eviction at the NUMA level + items: + description: NumaEvictionRankingMetric is the metrics + used to rank pods for eviction at the NUMA level + enum: + - qos.pod + - priority.pod + - mem.total.numa.container + type: string + minItems: 1 + type: array + numaFreeBelowWatermarkTimesThreshold: + description: NumaFreeBelowWatermarkTimesThreshold is the + threshold for the number of times NUMA's free memory + falls below the watermark + type: integer + rssOveruseRateThreshold: + description: RSSOveruseRateThreshold is the threshold + for the rate of rss + type: number + systemEvictionRankingMetrics: + description: SystemEvictionRankingMetrics is the metrics + used to rank pods for eviction at the system level + items: + description: SystemEvictionRankingMetric is the metrics + used to rank pods for eviction at the system level + enum: + - qos.pod + - priority.pod + - mem.usage.container + type: string + minItems: 1 + type: array + systemKswapdRateExceedDurationThreshold: + description: SystemKswapdRateExceedDurationThreshold is + the threshold for the duration the kswapd reclaiming + rate exceeds the threshold + type: integer + systemKswapdRateThreshold: + description: NumaFreeBelowWatermarkTimesThreshold is the + threshold for the rate of kswapd reclaiming rate + type: integer + type: object + reclaimedResourcesEvictionConfig: + description: ReclaimedResourcesEvictionConfig is the config + for reclaimed resources' eviction + properties: + evictionThreshold: + additionalProperties: + type: number + description: EvictionThreshold eviction threshold rate + for reclaimed resources + type: object + gracePeriod: + description: GracePeriod is the grace period of reclaimed + resources' eviction + format: int64 + type: integer + type: object + type: object reclaimedResourceConfig: description: ReclaimedResourceConfig is a configuration for reclaim resource properties: + cpuHeadroomConfig: + description: CPUHeadroomConfig is a configuration for cpu + headroom + properties: + utilBasedConfig: + description: UtilBasedConfig is a config for utilization + based cpu headroom policy + properties: + enable: + description: Enable is a flag to enable utilization + based cpu headroom policy + type: boolean + maxHeadroomCapacityRate: + description: MaxHeadroomCapacityRate is the max headroom + capacity rate of cpu headroom to the total cpu capacity + of node + type: number + maxOversoldRate: + description: MaxOversoldRate is the max oversold rate + of cpu headroom to the actual size of reclaimed_cores + pool + type: number + maxReclaimedCoreUtilization: + description: MaxReclaimedCoreUtilization is the max + reclaimed core utilization of reclaimed_cores pool, + which is used to calculate the oversold cpu headroom, + if zero means no limit + type: number + targetReclaimedCoreUtilization: + description: TargetReclaimedCoreUtilization is the + target reclaimed core utilization to be used for + calculating the oversold cpu headroom + type: number + type: object + type: object enableReclaim: description: EnableReclaim is a flag to enable reclaim resource, if true, reclaim resource will be enabled, which means reclaim @@ -74,6 +264,44 @@ spec: colocation between reclaimed_cores pod and other pods, otherwise, reclaim resource will be disabled. type: boolean + memoryHeadroomConfig: + description: MemoryHeadroomConfig is a configuration for memory + headroom + properties: + utilBasedConfig: + description: MemoryHeadroomUtilBasedConfig is a config + for utilization based memory headroom policy + properties: + cacheBasedRatio: + description: CacheBasedRatio is the rate of cache + oversold, 0 means disable cache oversold + type: number + enable: + description: Enable is a flag to enable utilization + based memory headroom policy + type: boolean + freeBasedRatio: + description: FreeBasedRatio is the estimation of free + memory utilization, which can be used as system + buffer to oversold memory. + type: number + staticBasedCapacity: + description: StaticBasedCapacity is the static oversold + memory size by bytes + type: number + type: object + type: object + minReclaimedResourceForAllocate: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: MinReclaimedResourceForAllocate is a resource + reserved for reclaimed_cores pods，these resources will not + be used by shared_cores pods. + type: object minReclaimedResourceForReport: additionalProperties: anyOf: @@ -113,7 +341,7 @@ spec: type: object type: object ephemeralSelector: - description: EphemeralSelector is used to indicate the + description: EphemeralSelector is a selector for temporary use only properties: lastDuration: description: define the duration this configuration will last @@ -127,12 +355,18 @@ spec: type: array type: object nodeLabelSelector: - description: NodeLabelSelector select nodes to apply these configurations - if spec.labelKey no set, NodeLabelSelector must be empty, the label - selector must only include KatalystCustomConfig.spec .nodeLabelSelectorKey, - otherwise it will not be synced This field will be immutable after - it is initially set. + description: NodeLabelSelector select nodes to apply these configurations, + the priority and node label selector must be matched according to + KatalystCustomConfig.spec.nodeLabelSelectorAllowedKeyList, otherwise + it will not be synced. type: string + priority: + description: Priority is used by one node matched by NodeLabelSelector + of more than one configuration, and the higher priority will be + considered. The priority only be supported when NodeLabelSelector + set + format: int32 + type: integer revisionHistoryLimit: default: 3 description: RevisionHistoryLimit is the maximum number of revisions @@ -141,6 +375,8 @@ spec: currently applied Spec version. The default value is 3. format: int64 type: integer + required: + - config type: object status: properties: diff --git a/charts/katalyst/crds/config.katalyst.kubewharf.io_evictionconfigurations.yaml b/charts/katalyst/crds/config.katalyst.kubewharf.io_evictionconfigurations.yaml deleted file mode 100644 index 57d6be9..0000000 --- a/charts/katalyst/crds/config.katalyst.kubewharf.io_evictionconfigurations.yaml +++ /dev/null @@ -1,211 +0,0 @@ ---- -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.9.0 - creationTimestamp: null - name: evictionconfigurations.config.katalyst.kubewharf.io -spec: - group: config.katalyst.kubewharf.io - names: - kind: EvictionConfiguration - listKind: EvictionConfigurationList - plural: evictionconfigurations - shortNames: - - ec - singular: evictionconfiguration - scope: Namespaced - versions: - - additionalPrinterColumns: - - jsonPath: .metadata.creationTimestamp - name: AGE - type: date - - jsonPath: .spec.nodeLabelSelector - name: SELECTOR - type: string - - jsonPath: .spec.ephemeralSelector.nodeNames - name: NODES - type: string - - jsonPath: .spec.ephemeralSelector.lastDuration - name: DURATION - type: string - - jsonPath: .status.conditions[?(@.type=="Valid")].status - name: VALID - type: string - - jsonPath: .status.conditions[?(@.type=="Valid")].reason - name: REASON - type: string - - jsonPath: .status.conditions[?(@.type=="Valid")].message - name: MESSAGE - type: string - name: v1alpha1 - schema: - openAPIV3Schema: - description: EvictionConfiguration is the Schema for the configuration API - used by eviction - properties: - apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' - type: string - kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' - type: string - metadata: - type: object - spec: - description: EvictionConfigurationSpec defines the desired state of EvictionConfiguration - properties: - config: - description: Config is custom field for eviction configuration all - configuration crd should contain a field named with `config` - properties: - evictionPluginsConfig: - description: EvictionPluginsConfig is the config for all eviction - plugins - properties: - memoryEvictionPluginConfig: - description: MemoryEvictionPluginConfig is the config for - memory eviction plugin - properties: - enableNumaLevelDetection: - description: EnableNumaLevelDetection is whether to enable - numa-level detection - type: boolean - enableRssOveruseDetection: - description: EnableRSSOveruseDetection is whether to enable - pod-level rss overuse detection - type: boolean - enableSystemLevelDetection: - description: EnableSystemLevelDetection is whether to - enable system-level detection - type: boolean - gracePeriod: - description: GracePeriod is the grace period of memory - eviction - format: int64 - type: integer - numaEvictionRankingMetrics: - description: NumaEvictionRankingMetrics is the metrics - used to rank pods for eviction at the NUMA level - items: - type: string - type: array - numaFreeBelowWatermarkTimesThreshold: - description: NumaFreeBelowWatermarkTimesThreshold is the - threshold for the number of times NUMA's free memory - falls below the watermark - type: integer - rssOveruseRateThreshold: - description: RssOveruseRateThreshold is the threshold - for the rate of rss - type: number - systemEvictionRankingMetrics: - description: SystemEvictionRankingMetrics is the metrics - used to rank pods for eviction at the system level - items: - type: string - type: array - systemKswapdRateExceedTimesThreshold: - description: SystemKswapdRateExceedCountThreshold is the - threshold for the number of times the kswapd reclaiming - rate exceeds the threshold - type: integer - systemKswapdRateThreshold: - description: NumaFreeBelowWatermarkTimesThreshold is the - threshold for the rate of kswapd reclaiming rate - type: integer - type: object - reclaimedResourcesEvictionPluginConfig: - description: ReclaimedResourcesEvictionPluginConfig is the - config for reclaimed resources eviction plugin - properties: - evictionThreshold: - additionalProperties: - type: number - description: EvictionThreshold eviction threshold rate - for reclaimed resources - type: object - type: object - type: object - type: object - ephemeralSelector: - description: EphemeralSelector is used to indicate the - properties: - lastDuration: - description: define the duration this configuration will last - from creationTimestamp. must and only set when NodeNames already - set - type: string - nodeNames: - description: Specific nodes' name the configurations will be effected. - items: - type: string - type: array - type: object - nodeLabelSelector: - description: NodeLabelSelector select nodes to apply these configurations - if spec.labelKey no set, NodeLabelSelector must be empty, the label - selector must only include KatalystCustomConfig.spec .nodeLabelSelectorKey, - otherwise it will not be synced This field will be immutable after - it is initially set. - type: string - revisionHistoryLimit: - default: 3 - description: RevisionHistoryLimit is the maximum number of revisions - that will be maintained in the resource's revision history. The - revision history consists of all revisions not represented by a - currently applied Spec version. The default value is 3. - format: int64 - type: integer - type: object - status: - properties: - collisionCount: - description: Count of hash collisions for this cr. The kcc controller - uses this field as a collision avoidance mechanism when it needs - to create the name for the newest ControllerRevision. - format: int32 - type: integer - conditions: - description: Represents the latest available observations of a config's - current state. - items: - properties: - lastTransitionTime: - description: Last time the condition transit from one status - to another. - format: date-time - type: string - message: - description: message is a human-readable explanation containing - details about the transition - type: string - reason: - description: reason is the reason for the condition's last transition. - type: string - status: - description: Status of the condition, one of True, False, Unknown. - type: string - type: - description: Type of config condition - type: string - required: - - status - - type - type: object - type: array - observedGeneration: - description: The most recent generation observed by the kcc controller. - format: int64 - type: integer - type: object - type: object - served: true - storage: true - subresources: - status: {} diff --git a/charts/katalyst/crds/config.katalyst.kubewharf.io_katalystcustomconfigs.yaml b/charts/katalyst/crds/config.katalyst.kubewharf.io_katalystcustomconfigs.yaml index de87dcf..c4ef365 100644 --- a/charts/katalyst/crds/config.katalyst.kubewharf.io_katalystcustomconfigs.yaml +++ b/charts/katalyst/crds/config.katalyst.kubewharf.io_katalystcustomconfigs.yaml @@ -27,9 +27,6 @@ spec: - jsonPath: .spec.targetType.version name: VERSION type: string - - jsonPath: .spec.nodeLabelSelectorKey - name: KEY - type: string - jsonPath: .metadata.generation name: GENERATION type: integer @@ -74,10 +71,31 @@ spec: description: whether disable revisionHistory for the KatalystCustomConfig resource type: boolean - nodeLabelSelectorKey: - description: key of node label to select which nodes will be effected - by the KatalystCustomConfig resource - type: string + nodeLabelSelectorAllowedKeyList: + description: the keys list allowed in node selector to select which + nodes will be effected by the KatalystCustomConfig resource, and + the priority will be used when one node match two KatalystCustomConfig + resource at the same time, the higher priority one will be considered. + If not set, node label selector is not allowed to use. + items: + description: PriorityNodeLabelSelectorAllowedKeyList defines the + priority and its allowed key list + properties: + keyList: + description: KeyList is allowed to use in node selector in the + Priority + items: + type: string + type: array + priority: + description: Priority is the priority of configurations + format: int32 + type: integer + required: + - keyList + - priority + type: object + type: array targetType: description: the GVR of target config type properties: diff --git a/charts/katalyst/crds/node.katalyst.kubewharf.io_customnoderesources.yaml b/charts/katalyst/crds/node.katalyst.kubewharf.io_customnoderesources.yaml index 9a872d7..a0e47de 100644 --- a/charts/katalyst/crds/node.katalyst.kubewharf.io_customnoderesources.yaml +++ b/charts/katalyst/crds/node.katalyst.kubewharf.io_customnoderesources.yaml @@ -224,18 +224,55 @@ spec: pairs. type: object type: object + siblings: + description: Siblings represents the relationship between TopologyZones + at the same level; for instance, the distance between NUMA + nodes. + items: + description: Sibling describes the relationship between two + Zones. + properties: + attributes: + description: Attributes are the attributes of the relationship + between two Zones. For instance, the distance between + tow NUMA nodes, the connection type between two GPUs, + etc. + items: + description: Attribute records the resource-specified + info with name-value pairs + properties: + name: + type: string + value: + type: string + required: + - name + - value + type: object + type: array + name: + description: Name represents the name of this Sibling. + type: string + type: + description: Type represents the type of this Sibling. + For instance, Socket, Numa, GPU, NIC, Disk and so on. + type: string + required: + - name + - type + type: object + type: array type: description: Type represents which kind of resource this TopologyZone is for; for instance, Socket, Numa, GPU, NIC, Disk and so on. type: string required: - - resources + - name - type type: object type: array required: - - resources - topologyPolicy type: object type: object