MON-4029: Add KubeStateMetricsConfig to ClusterMonitoring API#2778
MON-4029: Add KubeStateMetricsConfig to ClusterMonitoring API#2778danielmellado wants to merge 1 commit intoopenshift:masterfrom
Conversation
Adds a new KubeStateMetricsConfig struct to the ClusterMonitoring CRD, allowing configuration of the kube-state-metrics agent (node selectors, resources, tolerations, topology spread constraints). Includes comprehensive integration tests for validation. Signed-off-by: Daniel Mellado <dmellado@redhat.com>
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
openshift-ci-robot
added
the
jira/valid-reference
|
@danielmellado: This pull request references MON-4029 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Hello @danielmellado! Some important instructions when contributing to openshift/api: |
openshift-ci
bot
added
the
size/XXL
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@everettraven this PR continues #2461 and rebases and adds your comments. Thanks! |
📝 WalkthroughWalkthroughA new optional ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.11.3)Error: build linters: unable to load custom analyzer "kubeapilinter": tools/_output/bin/kube-api-linter.so, plugin: not implemented Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@payload-manifests/crds/0000_10_config-operator_01_clustermonitorings.crd.yaml`:
- Around line 1117-1234: The CRD currently documents invariants but doesn't
enforce them; update the schema for the PodTopologySpread-like object to:
restrict whenUnsatisfiable to an enum of allowed values (e.g.,
"DoNotSchedule","ScheduleAnyway") on the whenUnsatisfiable property, add
minimum: 1 (and disallow 0) on maxSkew (format int32) and minimum: 1 on
minDomains (or use minimum: 1 when present), and add a validation rule that
requires labelSelector when matchLabelKeys is set (make matchLabelKeys mutually
exclusive with labelSelector and/or add a x-kubernetes-requirements or OpenAPI
dependency that enforces labelSelector presence when matchLabelKeys exists).
Target the properties named whenUnsatisfiable, maxSkew, minDomains,
matchLabelKeys and labelSelector in the CRD fragment to implement these
constraints.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro
Run ID: a91c322b-92fd-43a0-a96f-b2b4da86aa91
⛔ Files ignored due to path filters (3)
config/v1alpha1/zz_generated.crd-manifests/0000_10_config-operator_01_clustermonitorings.crd.yamlis excluded by!**/zz_generated.crd-manifests/*config/v1alpha1/zz_generated.featuregated-crd-manifests/clustermonitorings.config.openshift.io/ClusterMonitoringConfig.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**openapi/generated_openapi/zz_generated.openapi.gois excluded by!openapi/**
📒 Files selected for processing (5)
config/v1alpha1/tests/clustermonitorings.config.openshift.io/ClusterMonitoringConfig.yamlconfig/v1alpha1/types_cluster_monitoring.goconfig/v1alpha1/zz_generated.deepcopy.goconfig/v1alpha1/zz_generated.swagger_doc_generated.gopayload-manifests/crds/0000_10_config-operator_01_clustermonitorings.crd.yaml
| matchLabelKeys: | ||
| description: |- | ||
| MatchLabelKeys is a set of pod label keys to select the pods over which | ||
| spreading will be calculated. The keys are used to lookup values from the | ||
| incoming pod labels, those key-value labels are ANDed with labelSelector | ||
| to select the group of existing pods over which spreading will be calculated | ||
| for the incoming pod. The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. | ||
| MatchLabelKeys cannot be set when LabelSelector isn't set. | ||
| Keys that don't exist in the incoming pod labels will | ||
| be ignored. A null or empty list means only match against labelSelector. | ||
|
|
||
| This is a beta field and requires the MatchLabelKeysInPodTopologySpread feature gate to be enabled (enabled by default). | ||
| items: | ||
| type: string | ||
| type: array | ||
| x-kubernetes-list-type: atomic | ||
| maxSkew: | ||
| description: |- | ||
| MaxSkew describes the degree to which pods may be unevenly distributed. | ||
| When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference | ||
| between the number of matching pods in the target topology and the global minimum. | ||
| The global minimum is the minimum number of matching pods in an eligible domain | ||
| or zero if the number of eligible domains is less than MinDomains. | ||
| For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same | ||
| labelSelector spread as 2/2/1: | ||
| In this case, the global minimum is 1. | ||
| | zone1 | zone2 | zone3 | | ||
| | P P | P P | P | | ||
| - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; | ||
| scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) | ||
| violate MaxSkew(1). | ||
| - if MaxSkew is 2, incoming pod can be scheduled onto any zone. | ||
| When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence | ||
| to topologies that satisfy it. | ||
| It's a required field. Default value is 1 and 0 is not allowed. | ||
| format: int32 | ||
| type: integer | ||
| minDomains: | ||
| description: |- | ||
| MinDomains indicates a minimum number of eligible domains. | ||
| When the number of eligible domains with matching topology keys is less than minDomains, | ||
| Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. | ||
| And when the number of eligible domains with matching topology keys equals or greater than minDomains, | ||
| this value has no effect on scheduling. | ||
| As a result, when the number of eligible domains is less than minDomains, | ||
| scheduler won't schedule more than maxSkew Pods to those domains. | ||
| If value is nil, the constraint behaves as if MinDomains is equal to 1. | ||
| Valid values are integers greater than 0. | ||
| When value is not nil, WhenUnsatisfiable must be DoNotSchedule. | ||
|
|
||
| For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same | ||
| labelSelector spread as 2/2/2: | ||
| | zone1 | zone2 | zone3 | | ||
| | P P | P P | P P | | ||
| The number of domains is less than 5(MinDomains), so "global minimum" is treated as 0. | ||
| In this situation, new pod with the same labelSelector cannot be scheduled, | ||
| because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, | ||
| it will violate MaxSkew. | ||
| format: int32 | ||
| type: integer | ||
| nodeAffinityPolicy: | ||
| description: |- | ||
| NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector | ||
| when calculating pod topology spread skew. Options are: | ||
| - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations. | ||
| - Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations. | ||
|
|
||
| If this value is nil, the behavior is equivalent to the Honor policy. | ||
| type: string | ||
| nodeTaintsPolicy: | ||
| description: |- | ||
| NodeTaintsPolicy indicates how we will treat node taints when calculating | ||
| pod topology spread skew. Options are: | ||
| - Honor: nodes without taints, along with tainted nodes for which the incoming pod | ||
| has a toleration, are included. | ||
| - Ignore: node taints are ignored. All nodes are included. | ||
|
|
||
| If this value is nil, the behavior is equivalent to the Ignore policy. | ||
| type: string | ||
| topologyKey: | ||
| description: |- | ||
| TopologyKey is the key of node labels. Nodes that have a label with this key | ||
| and identical values are considered to be in the same topology. | ||
| We consider each <key, value> as a "bucket", and try to put balanced number | ||
| of pods into each bucket. | ||
| We define a domain as a particular instance of a topology. | ||
| Also, we define an eligible domain as a domain whose nodes meet the requirements of | ||
| nodeAffinityPolicy and nodeTaintsPolicy. | ||
| e.g. If TopologyKey is "kubernetes.io/hostname", each Node is a domain of that topology. | ||
| And, if TopologyKey is "topology.kubernetes.io/zone", each zone is a domain of that topology. | ||
| It's a required field. | ||
| type: string | ||
| whenUnsatisfiable: | ||
| description: |- | ||
| WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy | ||
| the spread constraint. | ||
| - DoNotSchedule (default) tells the scheduler not to schedule it. | ||
| - ScheduleAnyway tells the scheduler to schedule the pod in any location, | ||
| but giving higher precedence to topologies that would help reduce the | ||
| skew. | ||
| A constraint is considered "Unsatisfiable" for an incoming pod | ||
| if and only if every possible node assignment for that pod would violate | ||
| "MaxSkew" on some topology. | ||
| For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same | ||
| labelSelector spread as 3/1/1: | ||
| | zone1 | zone2 | zone3 | | ||
| | P P P | P | P | | ||
| If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled | ||
| to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies | ||
| MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler | ||
| won't make it *more* imbalanced. | ||
| It's a required field. | ||
| type: string | ||
| required: | ||
| - maxSkew | ||
| - topologyKey | ||
| - whenUnsatisfiable | ||
| type: object |
There was a problem hiding this comment.
Enforce documented topology constraint invariants in the schema
The descriptions document hard constraints, but the schema does not enforce several of them (whenUnsatisfiable allowed values, positive maxSkew/minDomains, and matchLabelKeys requiring labelSelector). Invalid CRs can pass admission and then fail later during reconciliation/pod creation.
🔧 Proposed schema validation additions
maxSkew:
description: |-
MaxSkew describes the degree to which pods may be unevenly distributed.
@@
format: int32
+ minimum: 1
type: integer
minDomains:
description: |-
MinDomains indicates a minimum number of eligible domains.
@@
format: int32
+ minimum: 1
type: integer
@@
whenUnsatisfiable:
description: |-
WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy
@@
+ enum:
+ - DoNotSchedule
+ - ScheduleAnyway
type: string
+ x-kubernetes-validations:
+ - message: minDomains can only be set when whenUnsatisfiable is DoNotSchedule
+ rule: '!has(self.minDomains) || self.whenUnsatisfiable == ''DoNotSchedule'''
+ - message: matchLabelKeys cannot be set when labelSelector is not set
+ rule: '!has(self.matchLabelKeys) || has(self.labelSelector)'
required:
- maxSkew
- topologyKey
- whenUnsatisfiableAs per coding guidelines, "Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@payload-manifests/crds/0000_10_config-operator_01_clustermonitorings.crd.yaml`
around lines 1117 - 1234, The CRD currently documents invariants but doesn't
enforce them; update the schema for the PodTopologySpread-like object to:
restrict whenUnsatisfiable to an enum of allowed values (e.g.,
"DoNotSchedule","ScheduleAnyway") on the whenUnsatisfiable property, add
minimum: 1 (and disallow 0) on maxSkew (format int32) and minimum: 1 on
minDomains (or use minimum: 1 when present), and add a validation rule that
requires labelSelector when matchLabelKeys is set (make matchLabelKeys mutually
exclusive with labelSelector and/or add a x-kubernetes-requirements or OpenAPI
dependency that enforces labelSelector presence when matchLabelKeys exists).
Target the properties named whenUnsatisfiable, maxSkew, minDomains,
matchLabelKeys and labelSelector in the CRD fragment to implement these
constraints.
|
@danielmellado: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Adds a new KubeStateMetricsConfig struct to the ClusterMonitoring CRD,
allowing configuration of the kube-state-metrics agent (node selectors,
resources, tolerations, topology spread constraints). Includes
comprehensive integration tests for validation.
Signed-off-by: Daniel Mellado dmellado@redhat.com