No terminations scheduled problem

[shreya8887]

Hello, I am facing similar issue which you already close it before #80. However, I am working on it from past few days and I am not able to solve it.

    I0721 19:41:07.985992       1 config.go:82] Successfully validated configs
    I0721 19:41:07.986015       1 main.go:54] Starting kube-monkey with v logging level 5 and local log directory /var/log/kube-monkey
    I0721 19:41:08.011412       1 kubemonkey.go:19] Debug mode detected!
    I0721 19:41:08.011452       1 kubemonkey.go:20] Status Update: Generating next schedule in 300 sec
    I0721 19:46:08.011598       1 schedule.go:64] Status Update: Generating schedule for terminations
    I0721 19:46:08.179514       1 schedule.go:57] Status Update: 0 terminations scheduled today
        ********** Today's schedule **********
    No terminations scheduled
        ********** End of schedule **********
    I0721 19:46:08.179589       1 kubemonkey.go:62] Status Update: Waiting to run scheduled terminations.
    I0721 19:46:08.179603       1 kubemonkey.go:76] Status Update: All terminations done.

Below are my config file for more details:

1. configmap.yaml

        ---
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: kube-monkey-config-map
            namespace: kube-system
          data:
            config.toml: |
              [kubemonkey]
              run_hour = 8
              start_hour = 10
              end_hour = 16
              grace period_sec=120
              blacklisted_namespaces = ["kube-system"]
              whitelisted_namespaces=["default"]
              time_zone="Canada/Eastern"
              [debug]
              enabled = true
              schedule_delay=300
              force_should_kill = true
              schedule_immediate_kill = true
   

2. deployment.yaml

        ---
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: kube-monkey
            namespace: kube-system
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: kube-monkey
            template:
              metadata:
                labels:
                  app: kube-monkey
              spec:
                serviceAccount:default
                serviceAccountName:default
                containers:
                  -  name: kube-monkey
                     command:
                       - "/kube-monkey"
                     args: ["-v=5", "-log_dir=/var/log/kube-monkey"]
                     image: ayushsobti/kube-monkey:v0.3.0
                     env:
                       - name: KUBE_MONKEY_ID
                         value: CLUSTER_A
                     volumeMounts:
                       - name: config-volume
                         mountPath: "/etc/kube-monkey"
                volumes:
                  - name: config-volume
                    configMap:
                      name: kube-monkey-config-map
   

3. docker-registry yaml(Victim application yaml)

          apiVersion: apps.openshift.io/v1
          kind: DeploymentConfig
          metadata:
            creationTimestamp: '2021-07-14T18:09:09Z'
            generation: 6
            labels:
              docker-registry: default
              kube-monkey/enabled: enabled
              kube-monkey/identifier: docker-registry
              kube-monkey/kill-mode: fixed
              kube-monkey/kill-value: '1'
              kube-monkey/mtbf: '1'
            name: docker-registry
            namespace: default
            resourceVersion: '313891'
            selfLink: >-
              /apis/apps.openshift.io/v1/namespaces/default/deploymentconfigs/docker-registry
            uid: 9604c68c-e4ce-11eb-bd40-08002774d158
          spec:
            replicas: 3
            revisionHistoryLimit: 10
            selector:
              docker-registry: default
            strategy:
              activeDeadlineSeconds: 21600
              resources: {}
              rollingParams:
                intervalSeconds: 1
                maxSurge: 25%
                maxUnavailable: 25%
                timeoutSeconds: 600
                updatePeriodSeconds: 1
              type: Rolling
            template:
              metadata:
                creationTimestamp: null
                labels:
                  docker-registry: default
                  kube-monkey/enabled: enabled
                  kube-monkey/identifier: docker-registry
                  kube-monkey/kill-mode: fixed
                  kube-monkey/kill-value: '1'
                  kube-monkey/mtbf: '1'
              spec:
                containers:
                  - env:
                      - name: REGISTRY_HTTP_ADDR
                        value: ':5000'
                      - name: REGISTRY_HTTP_NET
                        value: tcp
                      - name: REGISTRY_HTTP_SECRET
                        value: t84he+iqXzoG4l3QNS4Y02m4TxV2P7Ww53k/BvoNZKU=
                      - name: REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA
                        value: 'false'
                    image: 'openshift/origin-docker-registry:v3.11.0'
                    imagePullPolicy: IfNotPresent
                    livenessProbe:
                      failureThreshold: 3
                      httpGet:
                        path: /healthz
                        port: 5000
                        scheme: HTTP
                      initialDelaySeconds: 10
                      periodSeconds: 10
                      successThreshold: 1
                      timeoutSeconds: 5
                    name: registry
                    ports:
                      - containerPort: 5000
                        protocol: TCP
                    readinessProbe:
                      failureThreshold: 3
                      httpGet:
                        path: /healthz
                        port: 5000
                        scheme: HTTP
                      periodSeconds: 10
                      successThreshold: 1
                      timeoutSeconds: 5
                    resources:
                      requests:
                        cpu: 100m
                        memory: 256Mi
                    securityContext:
                      privileged: true
                    terminationMessagePath: /dev/termination-log
                    terminationMessagePolicy: File
                    volumeMounts:
                      - mountPath: /registry
                        name: registry-storage
                dnsPolicy: ClusterFirst
                restartPolicy: Always
                schedulerName: default-scheduler
                securityContext: {}
                serviceAccount: registry
                serviceAccountName: registry
                terminationGracePeriodSeconds: 30
                volumes:
                  - hostPath:
                      path: /var/lib/minishift/base/openshift.local.pv/registry
                      type: ''
                    name: registry-storage
            test: false
            triggers:
              - type: ConfigChange
          status:
            availableReplicas: 3
            conditions:
              - lastTransitionTime: '2021-07-20T20:07:16Z'
                lastUpdateTime: '2021-07-20T20:07:16Z'
                message: Deployment config has minimum availability.
                status: 'True'
                type: Available
              - lastTransitionTime: '2021-07-20T21:01:15Z'
                lastUpdateTime: '2021-07-20T21:01:18Z'
                message: replication controller "docker-registry-4" successfully rolled out
                reason: NewReplicationControllerAvailable
                status: 'True'
                type: Progressing
            details:
              causes:
                - type: ConfigChange
              message: config change
            latestVersion: 4
            observedGeneration: 6
            readyReplicas: 3
            replicas: 3
            unavailableReplicas: 0
            updatedReplicas: 3
   

I have also tried with same configmap.yaml that Kube-monkey repository has but hit the wall.

I have deploy Kube-monkey on OCP 4. It is using Kubernetes Version 1.19.0.
Note: I have also tried same thing with OpenShift 3.11.

Could anyone please help me with this problem ?

[worldtiki]

Hi @shreya8887! I'm not familiar with openshift so I'm not sure if I'll be able to help much.

But you can see here that kube-monkey only targets apps/v1 Deployments but in your case you are using a apps.openshift.io/v1 DeploymentConfig.

Does that create a regular deployment behind the scenes? And if so, does it carry over the labels?

[worldtiki]

It looks like someone tried to solve this before and that it requires code changes: faust64@6b9956d