Etcd can't start (failed to resolve)

[pservit]

Hi,
I'm trying to test piraeus on kops-created cluster (kubernetes 1.16.8)

After kubectl apply -f https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/deploy/all.yaml

etcd can't start

piraeus-controller-0       0/1     Init:0/1   0          9m34s
piraeus-csi-controller-0   5/5     Running    0          9m33s
piraeus-csi-controller-1   5/5     Running    0          9m33s
piraeus-csi-controller-2   5/5     Running    0          9m33s
piraeus-csi-node-4l9j6     2/2     Running    0          9m33s
piraeus-csi-node-dcdhf     2/2     Running    0          9m33s
piraeus-csi-node-gjxsn     2/2     Running    0          9m33s
piraeus-etcd-0             0/1     Running    6          9m34s
piraeus-etcd-1             0/1     Running    6          9m34s
piraeus-etcd-2             0/1     Error      6          9m34s
piraeus-node-hg7bs         0/1     Init:0/1   0          9m34s
piraeus-node-rxf6b         0/1     Init:0/1   0          9m34s
piraeus-node-vlh8b         0/1     Init:0/1   0          9m34s
piraeus-scaler-5bv8h       0/1     Init:0/1   0          9m34s
piraeus-scaler-cn2hw       0/1     Init:0/1   0          9m34s
piraeus-scaler-spvkr       0/1     Init:0/1   0          9m34s

Errors in etcd

{"level":"warn","ts":1590732738.2479806,"caller":"netutil/netutil.go:121","msg":"failed to resolve URL Host","url":"http://piraeus-etcd-0.piraeus-etcd:2380","host":"piraeus-etcd-0.piraeus-etcd:2380","retry-interval":1,"error":"lookup piraeus-etcd-0.piraeus-etcd on 100.64.0.10:53: no such host"}

[alexzhc]

Obviously, the core-dns or kube-dns in your k8s cluster does not work.

[gdchalloner]

I'm actually seeing this as well on GKE (v1.16.9). - I think the issue is likely one of:

• kube-dns doesn't seem to populate DNS for unready services even with the annotation set. This looks like it was also an issue for CoreDNS but is now marked as fixed.
• some people run with a lower DNS ndot values which preempts DNS hammering issues but breaks the more 'magical' kube dns resolution so it's probably better if the members are fully qualified (in the log message it looks like it's using the unqualified hostname).

[rsliotta]

Hello all. I am having a similar issue on K8s 1.16 with several of the services. I will speculate you are using Alpine containers which are known to have MANY resolver issues...

[alexzhc]

Hello all. I am having a similar issue on K8s 1.16 with several of the services. I will speculate you are using Alpine containers which are known to have MANY resolver issues...

Let me switch to a debian based image with fully qualified name