kubernetes:yaml/v2:ConfigGroup resource does not respect skipUpdateUnreachable provider flag

[rtrindvg]

What happened?

In a stack I had to recreate a network peering between networks, which make an AKS (k8s) cluster unreachable. With the skipUpdateUnreachable flag on the provider, all the corresponding elements where skipped properly (like the k8s namespace), but not the configGroup. This means I need to manually make the cluster reachable to run pulumi or edit the state to remove the config group until the cluster is reachable again.

Error below:

  pulumi:pulumi:Stack (pulumi-aks-data-stage):
    error: kubernetes:yaml/v2:ConfigGroup resource 'k8s-cfgmap-secret-provider-class' has a problem: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "https://<redacted>.azmk8s.io:443/openapi/v2?timeout=32s": dial tcp x.x.x.x:443: i/o timeout

  kubernetes:core/v1:Namespace (k8s-ns-ingress):
    warning: Cluster is unreachable but skipUpdateUnreachable flag is set to true, skipping...

Example

import * as pulumi from '@pulumi/pulumi'
import { listManagedClusterUserCredentials } from '@pulumi/azure-native/containerservice/v20240801'
import { Provider } from '@pulumi/kubernetes'
import { ConfigGroup } from '@pulumi/kubernetes/yaml/v2'
import { Namespace } from '@pulumi/kubernetes/core/v1'

// Simplifying here, consider the AKS cluster already created beforehand in another stack
const cfgRgName = '<rgName>'
const cfgClusterName = '<clusterName>'
const cfgKeyVaultName = '<keyVaultName>'
const cfgTenantId = '<tenantId>'

const creds = pulumi.all(
  [cfgClusterName, cfgRgName],
).apply(([clusterName, rgName]) => listManagedClusterUserCredentials({
  resourceGroupName: rgName,
  resourceName: clusterName,
}))

const encoded = creds.kubeconfigs[0].value
export const kubeconfig = encoded.apply((enc) => Buffer.from(enc, 'base64').toString())

const coreProvider = new Provider('provider', {
  kubeconfig,
  skipUpdateUnreachable: true,
})

const k8sCfgmapSecretProviderClass = new ConfigGroup('k8s-cfgmap-secret-provider-class', {
  objs: [
    {
      apiVersion: 'secrets-store.csi.x-k8s.io/v1',
      kind: 'SecretProviderClass',
      metadata: {
        namespace: 'ingress',
        name: 'azure-tls',
      },
      spec: {
        provider: 'azure',
        secretObjects: [{
          secretName: 'ingress-tls-csi',
          type: 'kubernetes.io/tls',
          data: [
            {
              objectName: 'auto-cert-wild-contoso-inc',
              key: 'tls.key',
            }, {
              objectName: 'auto-cert-wild-contoso-inc',
              key: 'tls.crt',
            },
          ],
        }],
        parameters: {
          usePodIdentity: 'false',
          useVMManagedIdentity: 'true',
          userAssignedIdentityID: '',
          keyvaultName: cfgKeyVaultName,
          objects: 'array:\n  - |\n    objectName: auto-cert-wild-contoso-inc\n    objectType: secret\n',
          tenantId: cfgTenantId,
        },
      },
    },
  ],
}, {
  provider: coreProvider,
})

const k8sNsIngress = new Namespace('k8s-ns-ingress', {
  metadata: {
    name: 'ingress',
  },
}, {
  provider: coreProvider,
})

Output of pulumi about

CLI
Version 3.147.0
Go Version go1.23.5
Go Compiler gc

Plugins
KIND NAME VERSION
resource azure-native 2.81.0
resource kubernetes 4.19.0
language nodejs 3.147.0

Host
OS ubuntu
Version 24.04
Arch x86_64

This project is written in nodejs: executable='/home/rtrind/.nvm/versions/node/v22.11.0/bin/node' version='v22.11.0'

Current Stack: organization/pulumi/aks-data-stage

TYPE URN
pulumi:pulumi:Stack urn:pulumi:aks-data-stage::pulumi::pulumi:pulumi:Stack::pulumi-aks-data-stage
pulumi:providers:azure-native urn:pulumi:aks-data-stage::pulumi::pulumi:providers:azure-native::default_2_81_0
pulumi:providers:azure-native urn:pulumi:aks-data-stage::pulumi::pulumi:providers:azure-native::prov-stage-sub
pulumi:providers:azure-native urn:pulumi:aks-data-stage::pulumi::pulumi:providers:azure-native::prov-shared-sub
azure-native:resources:ResourceGroup urn:pulumi:aks-data-stage::pulumi::azure-native:resources:ResourceGroup::rg-aks-data-stage-
azure-native:network:VirtualNetwork urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetwork::vnet-aks-data-stage-
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-global-aks-data-stage-to-vpn
azure-native:network:VirtualNetworkLink urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkLink::vnl-pl-blob-to-aks-data-stage
azure-native:network:VirtualNetworkLink urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkLink::vnl-pl-db-to-aks-data-stage
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-aks-data-stage-to-spoke-stage
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-global-aks-data-stage-to-dpa-dev
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-aks-data-stage-to-spoke-shared
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-hub-shared-to-aks-data-stage
azure-native:network:Subnet urn:pulumi:aks-data-stage::pulumi::azure-native:network:Subnet::snet-aks-data-stage
azure-native:network:VirtualNetworkLink urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkLink::vnl-pl-postgres-to-aks-data-stage
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-aks-data-stage-to-hub-shared
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-spoke-shared-to-aks-data-stage
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-spoke-stage-to-aks-data-stage
azure-native:containerservice/v20210501:ManagedCluster urn:pulumi:aks-data-stage::pulumi::azure-native:containerservice/v20210501:ManagedCluster::aks-data-stage-
azure-native:authorization:RoleAssignment urn:pulumi:aks-data-stage::pulumi::azure-native:authorization:RoleAssignment::role-aks-to-kv-certs
azure-native:network:VirtualNetworkLink urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkLink::vnl-aks-data-stage-to-vpn
azure-native:authorization:RoleAssignment urn:pulumi:aks-data-stage::pulumi::azure-native:authorization:RoleAssignment::role-aks-to-acr-pull
azure-native:keyvault:AccessPolicy urn:pulumi:aks-data-stage::pulumi::azure-native:keyvault:AccessPolicy::kvap-aks-stage
azure-native:authorization:RoleAssignment urn:pulumi:aks-data-stage::pulumi::azure-native:authorization:RoleAssignment::role-aks-to-snet-aks-dnsrc
azure-native:authorization:RoleAssignment urn:pulumi:aks-data-stage::pulumi::azure-native:authorization:RoleAssignment::cloud-data-reader-aks-cluster-user-role
azure-native:authorization:RoleAssignment urn:pulumi:aks-data-stage::pulumi::azure-native:authorization:RoleAssignment::role-aks-to-vnet-aks-pdnszc
azure-native:containerservice/v20240801:MaintenanceConfiguration urn:pulumi:aks-data-stage::pulumi::azure-native:containerservice/v20240801:MaintenanceConfiguration::aks-data-config-stage
pulumi:providers:kubernetes urn:pulumi:aks-data-stage::pulumi::pulumi:providers:kubernetes::provider
pulumi:providers:azure-native urn:pulumi:aks-data-stage::pulumi::pulumi:providers:azure-native::default_2_76_0
kubernetes:core/v1:Namespace urn:pulumi:aks-data-stage::pulumi::kubernetes:core/v1:Namespace::k8s-ns-ingress
kubernetes:helm.sh/v3:Release urn:pulumi:aks-data-stage::pulumi::kubernetes:helm.sh/v3:Release::csi-secrets-store
pulumi:providers:pulumi urn:pulumi:aks-data-stage::pulumi::pulumi:providers:pulumi::default
kubernetes:yaml/v2:ConfigGroup urn:pulumi:aks-data-stage::pulumi::kubernetes:yaml/v2:ConfigGroup::k8s-cfgmap-secret-provider-class
kubernetes:secrets-store.csi.x-k8s.io/v1:SecretProviderClass urn:pulumi:aks-data-stage::pulumi::kubernetes:yaml/v2:ConfigGroup$kubernetes:secrets-store.csi.x-k8s.io/v1:SecretProviderClass::k8s-cfgmap-secret-provider-class:ingress/azure-tls
kubernetes:helm.sh/v3:Release urn:pulumi:aks-data-stage::pulumi::kubernetes:helm.sh/v3:Release::nginx-ingress
azure-native:network:VirtualNetworkPeering urn:pulumi:aks-data-stage::pulumi::azure-native:network:VirtualNetworkPeering::peer-global-vpn-to-aks-data-stage

Found no pending operations associated with aks-data-stage

Backend
Name
URL azblob://pulumi-state?storage_account=
User
Organizations
Token type personal

Dependencies:
NAME VERSION
@pulumi/kubernetes 4.19.0
@pulumi/pulumi 3.145.0
typescript 5.7.2
@pulumi/azure-native 2.81.0

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

Thanks for reporting this issue and apologies that you're facing it.

I agree that all options for our Kubernetes provider should carry over to any of the yaml resources. We'll add this to our backlog to resolve.