config whitelist_namespaces required

[pawarmanoj]

I have configured kube-monkey in the kube-system namespace. The kube-monkey pod, deployment, replica-set and config are all seen in good health on the kubernetes dashboard.

$ sudo nano /etc/kube-monkey/config.toml
[kubemonkey]
dry_run = false
run_hour = 9
start_hour = 10
end_hour = 18
graceperiod_sec = 120
blacklisted_namespaces = ["kube-system","kube-chaos"]
time_zone = "Asia/Kolkata"
[debug]
enabled = true
schedule_delay=300
force_should_kill = true
schedule_immediate_kill = true

My deployment has the necessary labels to mark the pod for termination.
kube-chaos-tomcat-deployment.txt

Yet kube-monkey is unable to find any pods for termination.
Logs show :

I0703 18:16:03.623685 1 kubemonkey.go:19] Debug mode detected!
I0703 18:16:03.623705 1 kubemonkey.go:20] Status Update: Generating next schedule in 30 sec
I0703 18:16:33.623885 1 schedule.go:64] Status Update: Generating schedule for terminations
I0703 18:16:33.722343 1 schedule.go:57] Status Update: 0 terminations scheduled today
I0703 18:16:33.722398 1 kubemonkey.go:63] Status Update: Waiting to run scheduled terminations.
I0703 18:16:33.722410 1 kubemonkey.go:77] Status Update: All terminations done.

Could someone please assist me in debugging what my mistake could be?
Thanks.

[asobti]

@Spellchaser Could this be caused because no whitelisted_namespaces are specified?

[pawarmanoj]

Thanks a lot @asobti
I didn't realize that both the blacklisted_namespaces and whitelisted_namespaces would need to be specified for proper operation.
And is there a way to quickly refresh kube-monkey to pick up the updated config map? Currently I am deleting the existing kube-monkey pod, so the restarted pod picks up the updated config.

[Aergonus]

From beyond the Great Firewall, there is no way to restart easily. There's an open issue about the viper not following symlinks. I thought the configs should automatically put in a default whitelist all if it's not added, but I'd have to check the code.

…

On Wed, Jul 4, 2018, 1:58 PM pawarmanoj ***@***.***> wrote: Thanks a lot @asobti <https://github.com/asobti> I didn't realize that both the blacklisted_namespaces and whitelisted_namespaces would need to be specified for proper operation. And is there a way to quickly refresh kube-monkey to pick up the updated config map? Currently I am deleting the existing kube-monkey pod, so the restarted pod picks up the updated config. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACV_i-SVz7l3aQbWCkFW7OfU8FTwOB6Tks5uDFmZgaJpZM4VBcfl> .

[asobti]

@pawarmanoj Can you try adding whitelisted_namespaces = [] in your config file.

[asobti]

Regarding hot-reload of configs, as @Spellchaser mentions, its a known bug that requires an upstream fix in vip. See #17

[pawarmanoj]

Yes, after adding whitelisted_namespaces = [] in the config file, kube-monkey is successfully able to terminate the pods that have opted-in for termination. thanks again @Spellchaser and @asobti

[asobti]

@Spellchaser leaving this open for now until you can confirm whether needing to specify whitelisted_namespaces = [] to disable the whitelist is a bug or intended behavior

[Aergonus]

Definitely not intended behavior, not having it should work. Will check on Monday.

…

On Sat, Jul 7, 2018, 7:02 AM Ayush Sobti ***@***.***> wrote: @Spellchaser <https://github.com/Spellchaser> leaving this open for now until you can confirm whether needing to specify whitelisted_namespaces = [] to disable the whitelist is a bug or intended behavior — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACV_iwfS3gWxWxnBFzdWWkfnd9Vg0pTIks5uD-xugaJpZM4VBcfl> .

[Aergonus]

Found the issue: "namespace": "chaos-sandbox",
The default is to only whitelist the default namespace. It's listed in the docs, but it's confusing in the example.

[Aergonus]

Going to close this in favor of #5, opt-in whitelist