Skip to content
This repository has been archived by the owner on Dec 14, 2023. It is now read-only.

Operator crashes upon reconcilation if AWS resources are removed #25

Open
aldojimenez opened this issue Mar 8, 2022 · 0 comments
Open

Operator crashes upon reconcilation if AWS resources are removed #25

aldojimenez opened this issue Mar 8, 2022 · 0 comments

Comments

@aldojimenez
Copy link

Problem Statement

During my testing of this operator to ensure it's production readiness I encountered an issue where the operator would crash on reconcilation if the resources it had previously created were removed from AWS.

Steps to Reproduce

Create a resource:

apiVersion: irsa.voodoo.io/v1alpha1
kind: IamRoleServiceAccount
metadata:
  name: s3-get-lister 
spec:
  policy: 
    statement:
      - resource: "arn:aws:s3:::test-irsa-4gkut9fl"
        action:
          - "s3:Get*"
          - "s3:List*"

Ensure operator creates resource successfully:

2022-03-08T10:29:21.117Z	INFO	controllers.Aws	successfully created trust role policy (irsa-op-irsa-controlller-default-s3-get-lister) on aws
2022-03-08T10:29:21.552Z	INFO	controllers.Aws	successfully attached role (irsa-op-irsa-controlller-default-s3-get-lister) & policy (arn:aws:iam::703320575624:policy/irsa-operator/irsa-controlller/default/s3-get-lister/policy/irsa-op-irsa-controlller-default-s3-get-lister) on aws

Remove the created resources (note: the below resource names will vary depending on your cluster, operator name and namespace):

 aws iam detach-role-policy --policy-arn arn:aws:iam::<accountid>:policy/irsa-operator/irsa-controlller/default/s3-get-lister/policy/irsa-op-irsa-controlller-default-s3-get-lister --role-name irsa-op-irsa-controlller-default-s3-get-lister
 aws iam delete-policy --policy-arn arn:aws:iam::<accountid>:policy/irsa-operator/irsa-controlller/default/s3-get-lister/policy/irsa-op-irsa-controlller-default-s3-get-lister
 aws iam delete-role --role-name irsa-op-irsa-controlller-default-s3-get-lister

Now either make a modification to the resource and apply it:

apiVersion: irsa.voodoo.io/v1alpha1
kind: IamRoleServiceAccount
metadata:
  name: s3-get-lister 
spec:
  policy: 
    statement:
      - resource: "arn:aws:s3:::test-irsa-4gkut9fl"
        action:
          - "s3:Get*"
          - "s3:List*"
          - "s3:Put*"

or recreate the operator pod:

kubectl delete po -n irsa-controller irsa-controller-irsa-operator-7c87c75f4d-6lmtg

Expected Outcome

Operator pod should reconcile the changes and recreate the deleted resources. The operator pod logs should output the modifications made.

Actual Outcome

The operator is unable to reconcile the differences and the following output is constantly spammed by the pod:

2022-03-08T10:42:46.162Z	INFO	controllers.IamRoleServiceAccount	[default/s3-get-lister] : Failed to create policy : Operation cannot be fulfilled on policies.irsa.voodoo.io "s3-get-lister": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:42:46.261Z	INFO	controllers.IamRoleServiceAccount	[default/s3-get-lister] : Failed to create policy : Operation cannot be fulfilled on policies.irsa.voodoo.io "s3-get-lister": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:42:46.361Z	INFO	controllers.IamRoleServiceAccount	[default/s3-get-lister] : Failed to create policy : Operation cannot be fulfilled on policies.irsa.voodoo.io "s3-get-lister": the object has been modified; please apply your changes to the latest version and try again

When recreating the operator pod, it fails to start and outputs the following logs:

2022-03-08T10:47:37.386Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2022-03-08T10:47:37.386Z	INFO	setup	starting manager
I0308 10:47:37.386837       1 leaderelection.go:243] attempting to acquire leader lease irsa-controller/d8e70b98.voodoo.io...
2022-03-08T10:47:37.386Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I0308 10:47:54.856004       1 leaderelection.go:253] successfully acquired lease irsa-controller/d8e70b98.voodoo.io
2022-03-08T10:47:54.856Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"irsa-controller","name":"d8e70b98.voodoo.io","uid":"2555cf13-b658-4a04-9c96-ab7c7a35d8c9","apiVersion":"v1","resourceVersion":"1437116"}, "reason": "LeaderElection", "message": "irsa-controller-irsa-operator-7c87c75f4d-j9pnm_61841c80-e002-485c-aba7-346dc8f61461 became leader"}
2022-03-08T10:47:54.856Z	INFO	controller-runtime.manager.controller.role	Starting EventSource	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "Role", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.856Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Lease","namespace":"irsa-controller","name":"d8e70b98.voodoo.io","uid":"43ce5eba-2daf-49fd-8eb0-6f72c0b9887a","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1437117"}, "reason": "LeaderElection", "message": "irsa-controller-irsa-operator-7c87c75f4d-j9pnm_61841c80-e002-485c-aba7-346dc8f61461 became leader"}
2022-03-08T10:47:54.856Z	INFO	controller-runtime.manager.controller.iamroleserviceaccount	Starting EventSource	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.856Z	INFO	controller-runtime.manager.controller.policy	Starting EventSource	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "Policy", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.956Z	INFO	controller-runtime.manager.controller.role	Starting Controller	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "Role"}
2022-03-08T10:47:54.956Z	INFO	controller-runtime.manager.controller.role	Starting workers	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "Role", "worker count": 10}
2022-03-08T10:47:54.956Z	INFO	controller-runtime.manager.controller.policy	Starting Controller	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "Policy"}
2022-03-08T10:47:54.957Z	INFO	controller-runtime.manager.controller.policy	Starting workers	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "Policy", "worker count": 10}
2022-03-08T10:47:54.956Z	INFO	controller-runtime.manager.controller.iamroleserviceaccount	Starting EventSource	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.957Z	INFO	controller-runtime.manager.controller.iamroleserviceaccount	Starting EventSource	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.957Z	INFO	controller-runtime.manager.controller.iamroleserviceaccount	Starting EventSource	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:55.058Z	INFO	controller-runtime.manager.controller.iamroleserviceaccount	Starting Controller	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount"}
2022-03-08T10:47:55.058Z	INFO	controller-runtime.manager.controller.iamroleserviceaccount	Starting workers	{"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "worker count": 10}
2022-03-08T10:47:55.197Z	INFO	controllers.Aws	role not found  on aws
E0308 10:47:55.197815       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 290 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x18eb100, 0x282d970)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x89
panic(0x18eb100, 0x282d970)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).attachPolicyToRoleIfNeeded(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0xc0004469c0)
	/workspace/controllers/role_controller.go:192 +0x50a
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).reconcilerRoutine(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0x7, 0xc00048de50, 0xd, 0xc000132600)
	/workspace/controllers/role_controller.go:146 +0x185
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).Reconcile(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc00048de60, 0x7, 0xc00048de50, 0xd, 0xc000657620, 0x40a1ff, 0xc000030000, ...)
	/workspace/controllers/role_controller.go:87 +0x1a5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x1952fe0, 0xc000449e00)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298 +0x317
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x0)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x1d8e680, 0xc0006ea100)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000055750)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005c9f50, 0x1d538c0, 0xc0006575c0, 0xc0006ea101, 0xc00057dc80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000055750, 0x3b9aca00, 0x0, 0x1, 0xc00057dc80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00, 0x0, 0x1)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:213 +0x45e
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x151a3ca]

goroutine 290 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x18eb100, 0x282d970)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).attachPolicyToRoleIfNeeded(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0xc0004469c0)
	/workspace/controllers/role_controller.go:192 +0x50a
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).reconcilerRoutine(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0x7, 0xc00048de50, 0xd, 0xc000132600)
	/workspace/controllers/role_controller.go:146 +0x185
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).Reconcile(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc00048de60, 0x7, 0xc00048de50, 0xd, 0xc000657620, 0x40a1ff, 0xc000030000, ...)
	/workspace/controllers/role_controller.go:87 +0x1a5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x1952fe0, 0xc000449e00)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298 +0x317
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x0)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x1d8e680, 0xc0006ea100)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000055750)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005c9f50, 0x1d538c0, 0xc0006575c0, 0xc0006ea101, 0xc00057dc80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000055750, 0x3b9aca00, 0x0, 0x1, 0xc00057dc80)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00, 0x0, 0x1)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:213 +0x45e
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aldojimenez