Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

coredns, metrics, nginx and snapshot validation services are always singleStack #4776

Closed
manuelbuil opened this issue Sep 20, 2023 · 3 comments
Closed

coredns, metrics, nginx and snapshot validation services are always singleStack #4776

manuelbuil opened this issue Sep 20, 2023 · 3 comments
Assignees
@manuelbuil @ShylajaDevadiga
Milestone
v1.28.3+rke2r1

Comments

@manuelbuil
Copy link
Contributor

Environmental Info:
RKE2 Version:

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Describe the bug:

This bug is covered by a bigger bug: #4772

When we are in dualStack mode, all these services are SingleStack. We can manually change that but I think it makes more sense if both were dualStack automatically

Steps To Reproduce:

  • Installed RKE2:

Deploy rke2 in dualStack env and verify what is the IpFamilyPolicy. It is always SingleStack and therefore, the service has only one IP

Expected behavior:

Services in dualStack mode have two IP addresses (ipv4, ipv6)

Actual behavior:

Services in dualStack mode have one IP address (ipv4)

Additional context / logs:
@manuelbuil manuelbuil self-assigned this Sep 20, 2023
@manuelbuil manuelbuil mentioned this issue Sep 20, 2023
Merged
@brandond
Copy link
Member

Is traefik also an issue? It's a LoadBalancer service, not ClusterIP, but should probably also use the same policy.

@manuelbuil
Copy link
Contributor Author

Is traefik also an issue? It's a LoadBalancer service, not ClusterIP, but should probably also use the same policy.

In k3s, we are already using preferDualStack for traefik as default https://github.com/k3s-io/k3s/blob/master/manifests/traefik.yaml#L40-L41

@manuelbuil manuelbuil mentioned this issue Sep 21, 2023
Merged
@ShylajaDevadiga ShylajaDevadiga self-assigned this Oct 2, 2023
This was referenced Oct 5, 2023
Closed
Closed
Closed
@caroline-suse-rancher caroline-suse-rancher added this to the v1.28.3+rke2r1 milestone Oct 10, 2023
@ShylajaDevadiga
Copy link
Contributor

Validated using rke2 version v1.28.3-rc2+rke2r1 (0d0d0e4)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

Cluster Configuration:
3 server 1 agent

Config.yaml:

$ cat config.yaml 
node-ip: <IPv6>,192.168.28.120
token: <TOKEN>
write-kubeconfig-mode: 644
cluster-cidr: 2001:cafe:42:0::/56,10.42.0.0/16
service-cidr: 2001:cafe:42:1::/112,10.43.0.0/16
cni: calico

Steps to reproduce the issue and validate the fix

  1. Copy config.yaml
  2. Install rke2

Replication results:

Validation results:

ubuntu@ip-192-168-28-120:~$ rke2 -v
rke2 version v1.28.3-rc2+rke2r1 (0d0d0e4879fdf95254461e3a49224f75d7b2dc3d)
go version go1.20.10 X:boringcrypto

Services have ipv6 address, based on the order in node-ip flag in the config.yaml

$ kubectl get svc -A
NAMESPACE     NAME                                      TYPE        CLUSTER-IP             EXTERNAL-IP   PORT(S)         AGE
default       kubernetes                                ClusterIP   2001:cafe:42:1::1      <none>        443/TCP         12m
kube-system   rke2-coredns-rke2-coredns                 ClusterIP   2001:cafe:42:1::a      <none>        53/UDP,53/TCP   12m
kube-system   rke2-ingress-nginx-controller-admission   ClusterIP   2001:cafe:42:1::8649   <none>        443/TCP         11m
kube-system   rke2-metrics-server                       ClusterIP   2001:cafe:42:1::bd1    <none>        443/TCP         11m
kube-system   rke2-snapshot-validation-webhook          ClusterIP   2001:cafe:42:1::4417   <none>        443/TCP         11m

Services have PreferDualStack in dualstack mode

$ kubectl describe svc -n kube-system    |grep  -i family -A4 -B2
Selector:          app.kubernetes.io/instance=rke2-coredns,app.kubernetes.io/name=rke2-coredns,k8s-app=kube-dns
Type:              ClusterIP
IP Family Policy:  PreferDualStack
IP Families:       IPv6,IPv4
IP:                2001:cafe:42:1::a
IPs:               2001:cafe:42:1::a,10.43.0.10
Port:              udp-53  53/UDP
--
Selector:          app.kubernetes.io/component=controller,app.kubernetes.io/instance=rke2-ingress-nginx,app.kubernetes.io/name=rke2-ingress-nginx
Type:              ClusterIP
IP Family Policy:  PreferDualStack
IP Families:       IPv6,IPv4
IP:                2001:cafe:42:1::8649
IPs:               2001:cafe:42:1::8649,10.43.174.100
Port:              https-webhook  443/TCP
--
Selector:          app=rke2-metrics-server,release=rke2-metrics-server
Type:              ClusterIP
IP Family Policy:  PreferDualStack
IP Families:       IPv6,IPv4
IP:                2001:cafe:42:1::bd1
IPs:               2001:cafe:42:1::bd1,10.43.25.138
Port:              <unset>  443/TCP
--
Selector:          app.kubernetes.io/instance=rke2-snapshot-validation-webhook,app.kubernetes.io/name=rke2-snapshot-validation-webhook
Type:              ClusterIP
IP Family Policy:  PreferDualStack
IP Families:       IPv6,IPv4
IP:                2001:cafe:42:1::4417
IPs:               2001:cafe:42:1::4417,10.43.147.76
Port:              https  443/TCP

Pods have ipv6 as well as ipv4 IP

...
IPs:
  IP:           192.168.23.28
  IP:           <IPv6 REDACTED>
Controlled By:  DaemonSet/rke2-canal
--
IPs:
  IP:           <IPv6> 
  IP:           192.168.2.244
Controlled By:  DaemonSet/rke2-canal
--
                      cni.projectcalico.org/podIPs: 10.42.1.2/32,2001:cafe:42:2::2/128
Status:               Running
IP:                   2001:cafe:42:2::2
IPs:
  IP:           2001:cafe:42:2::2
  IP:           10.42.1.2
Controlled By:  ReplicaSet/rke2-coredns-rke2-coredns-6b795db654
--
                      cni.projectcalico.org/podIPs: 10.42.0.4/32,2001:cafe:42::4/128
Status:               Running
IP:                   2001:cafe:42::4
IPs:
  IP:           2001:cafe:42::4
  IP:           10.42.0.4
Controlled By:  ReplicaSet/rke2-coredns-rke2-coredns-6b795db654
--
                      cni.projectcalico.org/podIPs: 10.42.0.2/32,2001:cafe:42::2/128
                      scheduler.alpha.kubernetes.io/tolerations: [{"key":"CriticalAddonsOnly", "operator":"Exists"}]
Status:               Running
IP:                   2001:cafe:42::2
IPs:
  IP:           2001:cafe:42::2
  IP:           10.42.0.2
Controlled By:  ReplicaSet/rke2-coredns-rke2-coredns-autoscaler-945fbd459
--
                  cni.projectcalico.org/podIPs: 10.42.1.3/32,2001:cafe:42:2::3/128
Status:           Running
IP:               2001:cafe:42:2::3
IPs:
  IP:           2001:cafe:42:2::3
  IP:           10.42.1.3
Controlled By:  DaemonSet/rke2-ingress-nginx-controller
--
                  cni.projectcalico.org/podIPs: 10.42.3.2/32,2001:cafe:42:4::2/128
Status:           Running
IP:               2001:cafe:42:4::2
IPs:
  IP:           2001:cafe:42:4::2
  IP:           10.42.3.2
Controlled By:  DaemonSet/rke2-ingress-nginx-controller
--
                  cni.projectcalico.org/podIPs: 10.42.2.2/32,2001:cafe:42:3::2/128
Status:           Running
IP:               10.42.2.2
IPs:
  IP:           10.42.2.2
  IP:           2001:cafe:42:3::2
Controlled By:  DaemonSet/rke2-ingress-nginx-controller
--
                  cni.projectcalico.org/podIPs: 10.42.0.13/32,2001:cafe:42::d/128
Status:           Running
IP:               2001:cafe:42::d
IPs:
  IP:           2001:cafe:42::d
  IP:           10.42.0.13
Controlled By:  DaemonSet/rke2-ingress-nginx-controller
--
                      cni.projectcalico.org/podIPs: 10.42.0.10/32,2001:cafe:42::a/128
Status:               Running
IP:                   2001:cafe:42::a
IPs:
  IP:           2001:cafe:42::a
  IP:           10.42.0.10
Controlled By:  ReplicaSet/rke2-metrics-server-544c8c66fc
--
                  cni.projectcalico.org/podIPs: 10.42.0.12/32,2001:cafe:42::c/128
Status:           Running
IP:               2001:cafe:42::c
IPs:
  IP:           2001:cafe:42::c
  IP:           10.42.0.12
Controlled By:  ReplicaSet/rke2-snapshot-controller-59cc9cd8f4
--
                  cni.projectcalico.org/podIPs: 10.42.0.9/32,2001:cafe:42::9/128
Status:           Running
IP:               2001:cafe:42::9
IPs:
  IP:           2001:cafe:42::9
  IP:           10.42.0.9
Controlled By:  ReplicaSet/rke2-snapshot-validation-webhook-54c5989b65

Pods display ipv6 IP

$ kubectl get pods -A -o wide
NAMESPACE     NAME                                                   READY   STATUS      RESTARTS      AGE   IP                                       NODE               NOMINATED NODE   READINESS GATES
kube-system   cloud-controller-manager-ip-192-168-18-33              1/1     Running     0             27m   2600:<IPv6>   ip-192-168-18-33   <none>           <none>
kube-system   cloud-controller-manager-ip-192-168-2-244              1/1     Running     0             19m   2600:<IPv6>    ip-192-168-2-244   <none>           <none>
kube-system   cloud-controller-manager-ip-192-168-9-18               1/1     Running     0             24m   2600:<IPv6>    ip-192-168-9-18    <none>           <none>
kube-system   etcd-ip-192-168-18-33                                  1/1     Running     0             27m   2600:<IPv6>   ip-192-168-18-33   <none>           <none>
kube-system   etcd-ip-192-168-2-244                                  1/1     Running     0             18m   2600:<IPv6>    ip-192-168-2-244   <none>           <none>
kube-system   etcd-ip-192-168-9-18                                   1/1     Running     0             23m   2600:<IPv6>   ip-192-168-9-18    <none>           <none>
kube-system   helm-install-rke2-canal-68dcd                          0/1     Completed   0             27m   2600:<IPv6>    ip-192-168-18-33   <none>           <none>
kube-system   helm-install-rke2-coredns-jl2bz                        0/1     Completed   0             27m   2600:<IPv6>    ip-192-168-18-33   <none>           <none>
kube-system   helm-install-rke2-ingress-nginx-7x2hj                  0/1     Completed   0             27m   2001:cafe:42::5                          ip-192-168-18-33   <none>           <none>
kube-system   helm-install-rke2-metrics-server-sr9vc                 0/1     Completed   0             27m   2001:cafe:42::8                          ip-192-168-18-33   <none>           <none>
kube-system   helm-install-rke2-snapshot-controller-crd-g68z4        0/1     Completed   0             27m   2001:cafe:42::6                          ip-192-168-18-33   <none>           <none>
kube-system   helm-install-rke2-snapshot-controller-zpkbk            0/1     Completed   1             27m   2001:cafe:42::7                          ip-192-168-18-33   <none>           <none>
kube-system   helm-install-rke2-snapshot-validation-webhook-mbksj    0/1     Completed   0             27m   2001:cafe:42::3                          ip-192-168-18-33   <none>           <none>
kube-system   kube-apiserver-ip-192-168-18-33                        1/1     Running     0             27m   2600:<IPv6>    ip-192-168-18-33   <none>           <none>
kube-system   kube-apiserver-ip-192-168-2-244                        1/1     Running     0             18m   2600:<IPv6>    ip-192-168-2-244   <none>           <none>
kube-system   kube-apiserver-ip-192-168-9-18                         1/1     Running     0             24m   2600:<IPv6>    ip-192-168-9-18    <none>           <none>
kube-system   kube-controller-manager-ip-192-168-18-33               1/1     Running     0             27m   2600:<IPv6>    ip-192-168-18-33   <none>           <none>
kube-system   kube-controller-manager-ip-192-168-2-244               1/1     Running     0             19m   2600:<IPv6>   ip-192-168-2-244   <none>           <none>
kube-system   kube-controller-manager-ip-192-168-9-18                1/1     Running     0             24m   2600:<IPv6>    ip-192-168-9-18    <none>           <none>
kube-system   kube-proxy-ip-192-168-18-33                            1/1     Running     0             27m   2600:<IPv6>    ip-192-168-18-33   <none>           <none>
kube-system   kube-proxy-ip-192-168-2-244                            1/1     Running     0             19m   2600:<IPv6>   ip-192-168-2-244   <none>           <none>
kube-system   kube-proxy-ip-192-168-23-28                            1/1     Running     0             20m   192.168.23.28                            ip-192-168-23-28   <none>           <none>
kube-system   kube-proxy-ip-192-168-9-18                             1/1     Running     0             24m   2600:<IPv6>    ip-192-168-9-18    <none>           <none>
kube-system   kube-scheduler-ip-192-168-18-33                        1/1     Running     0             27m   2600:<IPv6>   ip-192-168-18-33   <none>           <none>
kube-system   kube-scheduler-ip-192-168-2-244                        1/1     Running     0             19m   2600:<IPv6>   ip-192-168-2-244   <none>           <none>
kube-system   kube-scheduler-ip-192-168-9-18                         1/1     Running     0             24m   2600:<IPv6>   ip-192-168-9-18    <none>           <none>
kube-system   rke2-canal-566v9                                       2/2     Running     0             24m   2600:<IPv6>    ip-192-168-9-18    <none>           <none>
kube-system   rke2-canal-fqm4b                                       2/2     Running     0             27m   2600:<IPv6>   ip-192-168-18-33   <none>           <none>
kube-system   rke2-canal-jllhd                                       2/2     Running     0             20m   192.168.23.28                            ip-192-168-23-28   <none>           <none>
kube-system   rke2-canal-w2znz                                       2/2     Running     1 (18m ago)   19m   2600:<IPv6>   ip-192-168-2-244   <none>           <none>
kube-system   rke2-coredns-rke2-coredns-6b795db654-fqn2g             1/1     Running     0             24m   2001:cafe:42:2::2                        ip-192-168-9-18    <none>           <none>
kube-system   rke2-coredns-rke2-coredns-6b795db654-wq98z             1/1     Running     0             27m   2001:cafe:42::4                          ip-192-168-18-33   <none>           <none>
kube-system   rke2-coredns-rke2-coredns-autoscaler-945fbd459-5wlvh   1/1     Running     0             27m   2001:cafe:42::2                          ip-192-168-18-33   <none>           <none>
kube-system   rke2-ingress-nginx-controller-j9hgm                    1/1     Running     0             23m   2001:cafe:42:2::3                        ip-192-168-9-18    <none>           <none>
kube-system   rke2-ingress-nginx-controller-l8g55                    1/1     Running     0             18m   2001:cafe:42:4::2                        ip-192-168-2-244   <none>           <none>
kube-system   rke2-ingress-nginx-controller-m959n                    1/1     Running     0             19m   10.42.2.2                                ip-192-168-23-28   <none>           <none>
kube-system   rke2-ingress-nginx-controller-wt8dv                    1/1     Running     0             26m   2001:cafe:42::d                          ip-192-168-18-33   <none>           <none>
kube-system   rke2-metrics-server-544c8c66fc-tdzzp                   1/1     Running     0             26m   2001:cafe:42::a                          ip-192-168-18-33   <none>           <none>
kube-system   rke2-snapshot-controller-59cc9cd8f4-s7z6r              1/1     Running     0             26m   2001:cafe:42::c                          ip-192-168-18-33   <none>           <none>
kube-system   rke2-snapshot-validation-webhook-54c5989b65-hjnzx      1/1     Running     0             26m   2001:cafe:42::9                          ip-192-168-18-33   <none>           <none>
ubuntu@ip-192-168-18-33:~$

Validated pod to pod communication

$ kubectl exec -it multitool-deployment-564b975b9c-vmshh bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-5.1# ping 2001:cafe:42:4::3  
PING 2001:cafe:42:4::3(2001:cafe:42:4::3) 56 data bytes
64 bytes from 2001:cafe:42:4::3: icmp_seq=1 ttl=62 time=0.506 ms
64 bytes from 2001:cafe:42:4::3: icmp_seq=2 ttl=62 time=0.435 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@manuelbuil manuelbuil

@ShylajaDevadiga ShylajaDevadiga

Labels
None yet
Projects
None yet
Milestone
v1.28.3+rke2r1
Development

No branches or pull requests
4 participants
@brandond @manuelbuil @ShylajaDevadiga @caroline-suse-rancher