[OKD-SCOS v4.16] Machinesets/machines provisioned on Openstack(2024.1 Caracal (SLURP)) does not get deleted #2027

Grommingen · 2024-09-12T09:58:43Z

Grommingen
Sep 12, 2024

./openshift-install 4.16.0-0.okd-scos-2024-08-21-155613
built from commit deb993e72a920fb1c68a578d1b4c598071fefea4
release image quay.io/okd/scos-release@sha256:2d20d7770f511ca89d92b37812fdab66483d3d8cc2b1e9fcd33a341350e4f827
release architecture amd64

Hey.

We are in the process of testing machinesets for OKD SCOS on our Openstack installation and have noticed an issue in where the machines get stuck in a phase if you send in a faulty configuration parameter in a yaml file containing configuration of a given machineset. What we are trying to accomplish is to use an RBAC bridged network so that OKD nodes can live in different network segments. Along the way i've tried several configuration parameters, it is a bit hard to get a grasp of what actually gets accepted in the openstack providerSpec.

Applying a machineset:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk 
    machine.openshift.io/cluster-api-machine-role: apprunner
    machine.openshift.io/cluster-api-machine-type: apprunner
  name: apprunner
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk
      machine.openshift.io/cluster-api-machineset: apprunner 
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk
        machine.openshift.io/cluster-api-machine-role: apprunner
        machine.openshift.io/cluster-api-machine-type: apprunner
        machine.openshift.io/cluster-api-machineset: apprunner 
    spec:
      metadata:
       labels:
         node-role.kubernetes.io/apprunner: ""
      taints:
      - effect: NoSchedule
        key: node-role.kubernetes.io/apprunner
        value: reserved
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: gXcd.small
          image: okd-sandbox-jbhhk-rhcos
          kind: OpenstackProviderSpec
          networks: 
          - filter: {}
            subnets:
            - filter:
              # This subnet is / is not available so this is why it fails / succeeds
                id: <Input available / unavailable subnet>
                tags: openshiftClusterID=okd-sandbox-jbhhk
          securityGroups:
          - uuid: <uuid> 
          - uuid: <uuid>
          serverGroupName: apprunner
          serverMetadata:
            Name: apprunner
            openshiftClusterID: okd-sandbox-jbhhk
          tags:
          - openshiftClusterID=okd-sandbox-jbhhk 
          trunk: true
          userDataSecret:
            name: worker-user-data 
          availabilityZone: osl3

[core@okd-sandbox-jbhhk-master-2 ~]$ oc get machine
NAME                               PHASE         TYPE          REGION   ZONE   AGE
apprunner-hqs9r                    Provisioned   gXcd.small    osl      osl3   94s

Applying a non working machineset:
changing to a subnet that is not currently working

[core@okd-sandbox-jbhhk-master-2 ~]$ oc get machines
NAME                               PHASE          TYPE          REGION   ZONE   AGE
apprunner-hhp24                    Provisioning                                 25s

oc describe machine apprunner-hhp24

  Warning  Failedcreateport  22s (x14 over 100s)  openstack_controller  Failed to create port apprunner-hhp24-0: Request forbidden: [POST https://cloud.rlnc.eu/api/network/v2.0/ports], error message: {"NeutronError": {"type": "PolicyNotAuthorized", "message": "((rule:create_port and (rule:create_port:fixed_ips and (rule:create_port:fixed_ips:subnet_id))) and (rule:create_port:allowed_address_pairs and (rule:create_port:allowed_address_pairs:ip_address and rule:create_port:allowed_address_pairs:ip_address))) is disallowed by policy", "detail": ""}}

It does not show in openstack GUI. Since the config is not working i want to delete it again.

oc delete machineset apprunner-hhp24

The machineset and nodes gets deleted.

Applying another non working machineset that uses wrong config parameters:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk 
    machine.openshift.io/cluster-api-machine-role: apprunner
    machine.openshift.io/cluster-api-machine-type: apprunner
  name: apprunner
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk
      machine.openshift.io/cluster-api-machineset: apprunner 
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk
        machine.openshift.io/cluster-api-machine-role: apprunner
        machine.openshift.io/cluster-api-machine-type: apprunner
        machine.openshift.io/cluster-api-machineset: apprunner 
    spec:
      metadata:
       labels:
         node-role.kubernetes.io/apprunner: ""
      taints:
      - effect: NoSchedule
        key: node-role.kubernetes.io/apprunner
        value: reserved
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: gXcd.small
          image: okd-sandbox-jbhhk-rhcos
          kind: OpenstackProviderSpec
          network:
             id: <available/unavailable ID>
          securityGroups:
          #- filter: {}
          - uuid: <uuid>
          - uuid: <uuid>
          serverGroupName: apprunner
          serverMetadata:
            Name: apprunner
            openshiftClusterID: okd-sandbox-jbhhk
          tags:
          - openshiftClusterID=okd-sandbox-jbhhk 
          trunk: true
          userDataSecret:
            name: worker-user-data 
          availabilityZone: osl3

Logs from oc logs -f machine-api-controllers-6d9d4f8b8b-lh94s -c machine-controller shows:

I0912 08:45:16.679819       1 controller.go:115]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller"="machine-controller" "name"="apprunner-vkghn" "namespace"="openshift-machine-api" "object"={"name":"apprunner-vkghn","namespace":"openshift-machine-api"} "reconcileID"="ad4cd19c-2dd8-4ec9-a5e9-8e1514656059"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x23f4b55]

I want to delete this machineset then.

[core@okd-sandbox-jbhhk-master-1 .kube]$ oc delete machineset apprunner
machineset.machine.openshift.io "apprunner" deleted
[core@okd-sandbox-jbhhk-master-1 .kube]$ oc get machines
NAME                               PHASE          TYPE          REGION   ZONE   AGE
apprunner-vkghn                    Provisioning                                 12m

Stuck in provisioning.

[core@okd-sandbox-jbhhk-master-1 .kube]$ oc delete machines apprunner-vkghn
machine.machine.openshift.io "apprunner-vkghn" deleted

CLI unresponsive. But it gets deleted and the prompt resumes after a while. It is tempting to send a sigint(ctrl+C) but if you do not wait the machines will be stuck in the phase they are in i think.

Applying a completely borked machineset:

kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk 
    machine.openshift.io/cluster-api-machine-role: apprunner
    machine.openshift.io/cluster-api-machine-type: apprunner
  name: apprunner
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk
      machine.openshift.io/cluster-api-machineset: apprunner 
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: okd-sandbox-jbhhk
        machine.openshift.io/cluster-api-machine-role: apprunner
        machine.openshift.io/cluster-api-machine-type: apprunner
        machine.openshift.io/cluster-api-machineset: apprunner 
    spec:
      metadata:
       labels:
         node-role.kubernetes.io/apprunner: ""
      taints:
      - effect: NoSchedule
        key: node-role.kubernetes.io/apprunner
        value: reserved
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: gXcd.small
          image: okd-sandbox-jbhhk-rhcos
          kind: OpenstackProviderSpec
          ports:
           networkID: <networkID>
          securityGroups:
          #- filter: {}
          - uuid: <uuid> 
          - uuid: <uuid>
          serverGroupName: apprunner
          serverMetadata:
            Name: apprunner
            openshiftClusterID: okd-sandbox-jbhhk
          tags:
          - openshiftClusterID=okd-sandbox-jbhhk 
          trunk: true
          userDataSecret:
            name: worker-user-data 
          availabilityZone: osl

It does not seem like the machines created from this machineset will ever get deleted. We've had one stuck for 8 days deleting now.

oc logs -f machine-api-controllers-6d9d4f8b8b-lh94s -c machine-controller shows:

I0912 09:22:56.532708       1 controller.go:156] apprunner-ds5ww: reconciling Machine
E0912 09:22:56.532993       1 controller.go:266] apprunner-ds5ww: failed to check if machine exists: Failed to get Machine Spec from Provider Spec: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field OpenstackProviderSpec.ports of type []v1alpha1.PortOpts
E0912 09:22:56.541629       1 controller.go:329]  "msg"="Reconciler error" "error"="Failed to get Machine Spec from Provider Spec: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field OpenstackProviderSpec.ports of type []v1alpha1.PortOpts" "controller"="machine-controller" "name"="apprunner-ds5ww" "namespace"="openshift-machine-api" "object"={"name":"apprunner-ds5ww","namespace":"openshift-machine-api"} "reconcileID"="d420f659-10e5-4f5d-ba7e-d22f8463f500

bshephar · 2024-09-12T15:22:28Z

bshephar
Sep 12, 2024

The panic isn't great. Might want to understand what nil pointer is being dereferenced there. But as for the issue of deleting the bad VMs:

I can reproduce your issue easily enough:

$ oc logs -n openshift-machine-api deployment/machine-api-controllers -c machine-controller | tail -n4
E0912 15:09:22.215817       1 controller.go:329]  "msg"="Reconciler error" "error"="failed to get OpenStackProviderSpec from machineset: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field OpenstackProviderSpec.ports of type []v1alpha1.PortOpts" "MachineSet"={"name":"okd-k8pfr-worker-1","namespace":"openshift-machine-api"} "controller"="machineset" "controllerGroup"="machine.openshift.io" "controllerKind"="MachineSet" "name"="okd-k8pfr-worker-1" "namespace"="openshift-machine-api" "reconcileID"="f0bd1a52-5a8c-4b2d-9801-1484f8928964"
I0912 15:10:21.175188       1 controller.go:156] okd-k8pfr-worker-1-zmb5l: reconciling Machine
E0912 15:10:21.175543       1 controller.go:266] okd-k8pfr-worker-1-zmb5l: failed to check if machine exists: Failed to get Machine Spec from Provider Spec: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field OpenstackProviderSpec.ports of type []v1alpha1.PortOpts
E0912 15:10:21.184543       1 controller.go:329]  "msg"="Reconciler error" "error"="Failed to get Machine Spec from Provider Spec: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal object into Go struct field OpenstackProviderSpec.ports of type []v1alpha1.PortOpts" "controller"="machine-controller" "name"="okd-k8pfr-worker-1-zmb5l" "namespace"="openshift-machine-api" "object"={"name":"okd-k8pfr-worker-1-zmb5l","namespace":"openshift-machine-api"} "reconcileID"="188d836e-7e2b-44e0-a91f-a85301185482"

Since the VM won't exist in OpenStack, you could remove the Machines by patching the finalizers.

So, if I delete the MachineSet to get the machine stuck in Deleting:

[m3@osp-df-3 okd-4.16]$ oc delete -f machineset-worker
machineset.machine.openshift.io "okd-k8pfr-worker-1" deleted

[m3@osp-df-3 okd-4.16]$ oc get machine -n openshift-machine-api | grep Del
okd-k8pfr-worker-1-q7kz6   Deleting                                                          5m25s

Then I can just patch the finalizers to remove it:

[m3@osp-df-3 okd-4.16]$ oc patch -n openshift-machine-api machines/okd-k8pfr-worker-1-q7kz6 --type merge -p '{"metadata":{"finalizers":[]}}'
machine.machine.openshift.io/okd-k8pfr-worker-1-q7kz6 patched

[m3@osp-df-3 okd-4.16]$ oc get machine -n openshift-machine-api | grep Del
[m3@osp-df-3 okd-4.16]$

0 replies

bshephar · 2024-09-13T02:24:05Z

bshephar
Sep 13, 2024

I couldn't reproduce the panic. Is there more of a traceback in the logs? If you can maybe share more of the machine-controller logs with the full traceback it will be easier to investigate.

But it's probably an issue that needs to be raised against the machine-api-operator rather than OKD. I don't think this is going to be unique to OKD.

0 replies

Grommingen · 2024-09-13T07:54:44Z

Grommingen
Sep 13, 2024
Author

Applying with these options:

          network:
             id: available uuid
          securityGroups:
          #- filter: {}
          - uuid: available uuid
          - uuid: available uuid

Logs before pod crashes

I0913 07:51:09.579418       1 controller.go:51]  "msg"="Reconciling" "machineset"="apprunner" "namespace"="openshift-machine-api"
I0913 07:51:09.587739       1 utils.go:111] Cloud provider CA cert not provided, using system trust bundle
I0913 07:51:09.603587       1 controller.go:156] apprunner-z5xmg: reconciling Machine
I0913 07:51:09.612672       1 controller.go:156] apprunner-z5xmg: reconciling Machine
I0913 07:51:10.025918       1 controller.go:342] apprunner-z5xmg: setting phase to Provisioning and requeuing
I0913 07:51:10.025937       1 controller.go:404] apprunner-z5xmg: going into phase "Provisioning"
I0913 07:51:10.033286       1 controller.go:156] apprunner-z5xmg: reconciling Machine
I0913 07:51:10.083023       1 controller.go:51]  "msg"="Reconciling" "machineset"="apprunner" "namespace"="openshift-machine-api"
I0913 07:51:10.087477       1 utils.go:111] Cloud provider CA cert not provided, using system trust bundle
I0913 07:51:10.428435       1 controller.go:349] apprunner-z5xmg: reconciling machine triggers idempotent create
I0913 07:51:10.803792       1 utils.go:111] Cloud provider CA cert not provided, using system trust bundle
I0913 07:51:11.536878       1 utils.go:111] Cloud provider CA cert not provided, using system trust bundle
I0913 07:51:12.220708       1 controller.go:115]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller"="machine-controller" "name"="apprunner-z5xmg" "namespace"="openshift-machine-api" "object"={"name":"apprunner-z5xmg","namespace":"openshift-machine-api"} "reconcileID"="c43124d9-1052-4a44-96ed-14b2083bb397"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x23f4b55]

goroutine 236 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x2997a60?, 0x4093d30?})
	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute.(*Service).constructPorts(0xc000f0a930?, 0xc000df9250, 0xc000bf8a00)
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute/instance.go:188 +0xb5
sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute.(*Service).createInstanceImpl(0xc000df9208, {0x2ff8a48?, 0xc0008dcd80}, 0x2?, 0xc000bf8a00, {0xc000f0a960, 0x27}, 0x1?)
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute/instance.go:262 +0x26a
sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute.(*Service).CreateInstance(...)
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute/instance.go:225
github.com/openshift/machine-api-provider-openstack/pkg/machine.(*OpenstackClient).createInstance(0xc00026a7e0, {0xc000757980?, 0xc000f7ab10?}, 0xc0008dcd80, {0x3016800?, 0xc000757980})
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/pkg/machine/actuator.go:257 +0x2b4
github.com/openshift/machine-api-provider-openstack/pkg/machine.(*OpenstackClient).reconcile(0xc00026a7e0, {0x300f5a8, 0xc000f7ab10}, 0xc0008dcd80)
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/pkg/machine/actuator.go:200 +0x21e
github.com/openshift/machine-api-provider-openstack/pkg/machine.(*OpenstackClient).Create(0xc00090bc00?, {0x300f5a8?, 0xc000f7ab10?}, 0x0?)
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/pkg/machine/actuator.go:171 +0x1d
github.com/openshift/machine-api-operator/pkg/controller/machine.(*ReconcileMachine).Reconcile(0xc0004e2370, {0x300f5a8, 0xc000f7ab10}, {{{0xc000636df8?, 0x0?}, {0xc0006c8e80?, 0xc0006f1d48?}}})
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/github.com/openshift/machine-api-operator/pkg/controller/machine/controller.go:350 +0xba9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x3012b68?, {0x300f5a8?, 0xc000f7ab10?}, {{{0xc000636df8?, 0xb?}, {0xc0006c8e80?, 0x0?}}})
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000360dc0, {0x300f5e0, 0xc000524a00}, {0x2a87a80?, 0xc000004e20?})
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000360dc0, {0x300f5e0, 0xc000524a00})
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 182
	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565

oc get pods
NAME               READY       STATUS     RESTARTS        AGE
machine-api-controllers-6d9d4f8b8b-lh94s              7/7     Running   11 (2m2s ago)   23h

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OKD-SCOS v4.16] Machinesets/machines provisioned on Openstack(2024.1 Caracal (SLURP)) does not get deleted #2027

{{title}}

Replies: 3 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[OKD-SCOS v4.16] Machinesets/machines provisioned on Openstack(2024.1 Caracal (SLURP)) does not get deleted #2027

Grommingen Sep 12, 2024

Replies: 3 comments

bshephar Sep 12, 2024

bshephar Sep 13, 2024

Grommingen Sep 13, 2024 Author

Grommingen
Sep 12, 2024

bshephar
Sep 12, 2024

bshephar
Sep 13, 2024

Grommingen
Sep 13, 2024
Author