Reconciler fails to Patch at random

[sthaha]

Related to #181 .

We sometimes also see that reconciler fails to Patch. The issue seem to be that the Patch tries to update a non-existent resource when it should actually be creating it.

E.g. - https://github.com/rhobs/observability-operator/actions/runs/3232742256/jobs/5294131669#step:4:989

2022-10-12T07:58:40Z	INFO	observability-operator	Reconciling monitoring stack	{"stack": "e2e-tests/valid-loglevel"}
2022-10-12T07:58:40Z	INFO	thanos-querier	watched MonitoringStack changed, checking for matching querier	{"Monitoring Stack": "e2e-tests/valid-loglevel"}
2022-10-12T07:58:41Z	INFO	observability-operator	patch failed	{"stack": "e2e-tests/valid-loglevel", "reconciler": "updater", "name": "valid-loglevel-alertmanager", "ns": "e2e-tests", "error": "roles.rbac.authorization.k8s.io \"valid-loglevel-alertmanager\" not found"}
2022-10-12T07:58:41Z	INFO	observability-operator	Failed to get prometheus object	{"stack": "e2e-tests/valid-loglevel", "err": "Prometheus.monitoring.rhobs \"valid-loglevel\" not found"}
2022-10-12T07:58:41Z	ERROR	Reconciler error	{"controller": "monitoringstack", "controllerGroup": "monitoring.rhobs", "controllerKind": "MonitoringStack", "monitoringStack": {"name":"valid-loglevel","namespace":"e2e-tests"}, "namespace": "e2e-tests", "name": "valid-loglevel", "reconcileID": "45436353-5cee-45b0-947f-aefeed0e2d0a", "error": "roles.rbac.authorization.k8s.io \"valid-loglevel-alertmanager\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2022-10-12T07:58:41Z	INFO	observability-operator	Reconciling monitoring stack	{"stack": "e2e-tests/valid-loglevel"}

2022-10-12T07:58:51Z	INFO	observability-operator	Reconciling monitoring stack	{"stack": "e2e-tests/valid-retention"}
2022-10-12T07:58:51Z	INFO	thanos-querier	watched MonitoringStack changed, checking for matching querier	{"Monitoring Stack": "e2e-tests/valid-retention"}
2022-10-12T07:58:51Z	INFO	observability-operator	patch failed	{"stack": "e2e-tests/valid-retention", "reconciler": "updater", "name": "valid-retention-alertmanager", "ns": "e2e-tests", "error": "roles.rbac.authorization.k8s.io \"valid-retention-alertmanager\" not found"}
2022-10-12T07:58:51Z	INFO	observability-operator	Failed to get prometheus object	{"stack": "e2e-tests/valid-retention", "err": "Prometheus.monitoring.rhobs \"valid-retention\" not found"}
2022-10-12T07:58:51Z	ERROR	Reconciler error	{"controller": "monitoringstack", "controllerGroup": "monitoring.rhobs", "controllerKind": "MonitoringStack", "monitoringStack": {"name":"valid-retention","namespace":"e2e-tests"}, "namespace": "e2e-tests", "name": "valid-retention", "reconcileID": "6fd51d0c-50e2-4618-9107-5419858c5a50", "error": "roles.rbac.authorization.k8s.io \"valid-retention-alertmanager\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2022-10-12T07:58:51Z	INFO	observability-operator	Reconciling monitoring stack	{"stack": "e2e-tests/valid-retention"}
2022-10-12T07:58:51Z	INFO	observability-operator	Reconciling monitoring stack	{"stack": "e2e-tests/valid-retention"}