[Release-1.26] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5524

brandond · 2024-02-27T00:44:50Z

Backport fix for Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion

Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5516

endawkins · 2024-02-28T02:20:56Z

Validated on 1.26 with `9351840` / 1.26

Environment Details

Infrastructure

Cloud
Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-30-25 4.15.0-1051-aws #53-Ubuntu SMP Wed Sep 18 13:35:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Cluster Configuration:

4 Servers:
2 Bastion Hosts
2 Airgapped Instances

Config.yaml:

config.yaml 1:
token: test
debug: true
write-kubeconfig-mode: 644
system-default-registry: [REDACTED]

config.yaml 2:
token: test
debug: true
write-kubeconfig-mode: 644
system-default-registry: [REDACTED]

Additional files

registries.yaml
- airgap 1 [registry prefix] w/ uri schema
---
mirrors:
  docker.io:
    endpoint:
            - "https://[REDACTED]"
  [REDACTED]:
    endpoint:
            - "[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

airgap 1 [registry prefix] w/o uri schema

---
mirrors:
  docker.io:
    endpoint:
            - "[REDACTED]"
  [REDACTED]:
    endpoint:
            - "[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

registries.yaml w/o schema
- airgap 2 [no registry prefix]

---
mirrors:
  docker.io:
    endpoint:
      - "[REDACTED]"
  [REDACTED]:
    endpoint:
      - "[REDACTED]"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

Testing Steps

Note:

Bastion Instance 1 -> Airgap Instance 1
Bastion Instance 2 -> Airgap Instance 2

Air-Gap Setup

Launch two bastion instances from AWS
Launch two airgapped instances from AWS -- disable auto-assign public IP
ssh into the bastion nodes
Copy .pem file to bastion instances
scp -i "<path_to_pem_file>" <path_to_pem_file> username@<PUBLIC_IP>:~
ssh into airgapped instances

Pull-Through Cache Configuration

Add certificates: mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt -subj "/C=US/ST=AZ/O=Rancher QA/CN=[REDACTED]" -addext "subjectAltName = DNS:[REDACTED]"
Bastion 1:
sudo docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -e REGISTRY_HTTP_PREFIX=/testing/ -p 443:443 registry:2.7.1
Bastion 2:
sudo docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -p 443:443 registry:2.7.1
To view the containers running: docker ps -a

Copying Files to Airgapped Instance [Do this for each pair of airgapped instances]

Obtain the rke2 binary, rename it if desired (make sure its amd64): wget -O <NAME_THE_BINARY_FILE> https://github.com/rancher/rke2/releases/download/<VERSION>/<FILENAME>
ssh into airgapped instance
ssh -i <file_name>.pem username@<AIRGAP_IP>
close connection to airgapped instanceexit
copy rke2 binary and certificates to airgapped instance:

scp -i <file_name>.pem <rke2_binary_file> username@<AIRGAP_IP>:~
scp -i <file_name>.pem certs/* username@<AIRGAP_IP>:

ssh to airgapped instance

Certificates and RKE2 Setup

Update Certificates:
sudo cp domain.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
sudo vi config.yaml (there will be a config.yaml in both airgapped instances - a total of 2)
sudo vi registries (there will be a registries.yaml in both airgapped instances - a total of 2)
sudo mkdir -p /etc/rancher/rke2/ && sudo cp config.yaml /etc/rancher/rke2/ && cat /etc/rancher/rke2/config.yaml && cp registries.yaml /etc/rancher/rke2/ && cat /etc/rancher/rke2/registries.yaml
** Making RKE2 Binaries Executable **
chmod +x <RKE2_BINARY>
sudo mv <RKE2_BINARY> /usr/local/bin/rke2
Check version: rke2 --version
Open two new terminals and ssh into airgap instance 1 and 2

in those two terminals run the following command: sudo rke2 server

source .bashrc
kga
search for "?ns"

Replication Results:

rke2 version used for replication:

rke2 --version
rke2 version v1.26.13+rke2r1 (637e8a38334f603b60650b30547252a5c461fa0d)
go version go1.20.13 X:boringcrypto

Registries

registries.yaml
- airgap 1 [registry prefix] w/ uri schema
---
mirrors:
  docker.io:
    endpoint:
            - "https://[REDACTED]"
  [REDACTED]:
    endpoint:
            - "https://[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

airgap 1 [registry prefix] w/o uri schema

---
mirrors:
  docker.io:
    endpoint:
            - "[REDACTED]"
  [REDACTED]:
    endpoint:
            - "[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

registries.yaml w/o schema
- airgap 2 [no registry prefix]

---
mirrors:
  docker.io:
    endpoint:
      - "[REDACTED]"
  [REDACTED]:
    endpoint:
      - "[REDACTED]"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

Observations:

prefix + no uri schema:

Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/v2/?ns=[REDACTED]
W0227 22:57:05.649740    2954 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
WARN[0002] Failed to get image from endpoint: GET https://[REDACTED]/v2/?ns=REDACTED: unexpected status code 404 Not Found: 404 page not found
FATA[0002] failed to get runtime image [REDACTED]/rancher/rke2-runtime:v1.26.13-rke2r1: all endpoints failed: GET https://[REDACTED]/v2/?ns=REDACTED: unexpected status code 404 Not Found: 404 page not found

prefix + uri schema:

INFO[0002] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0002] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.26.13-rke2r1
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/testing/v2/?ns=[REDACTED]
W0227 22:58:16.196629    2966 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/v1.26.13-rke2r1 => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/v1.26.13-rke2r1?ns=[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED]?ns=[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED]?ns=[REDACTED]

Validation Results:

rke2 version used for validation:

rke2 -version
rke2 version v1.26.14-rc3+rke2r1 (93518400c62043756be7e321480bcd5f19f42bbf)
go version go1.21.7 X:boringcrypto

rke2 starts successfully with no ?ns=<registry_hostname> in the logs

Airgap 1:
INFO[0002] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0002] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.26.14-rc3-rke2r1
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/testing/v2/
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/v1.26.14-rc3-rke2r1 => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/v1.26.14-rc3-rke2r1
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED]


NAME                   STATUS   ROLES                       AGE    VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-30-25   Ready    control-plane,etcd,master   116s   v1.26.14+rke2r1   172.31.30.25   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws   containerd://1.7.11-k3s2

NAMESPACE     NAME                                                       READY   STATUS      RESTARTS   AGE    IP             NODE              NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-30-25               1/1     Running     0          112s   172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/etcd-ip-172-31-30-25                                   1/1     Running     0          88s    172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/helm-install-rke2-canal-klkfx                          0/1     Completed   0          98s    172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/helm-install-rke2-coredns-9ddfc                        0/1     Completed   0          98s    172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-t26nm                  0/1     Completed   0          98s    10.42.0.4      ip-172-31-30-25   <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-c4rkw                 0/1     Completed   0          98s    10.42.0.7      ip-172-31-30-25   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-xpkds        0/1     Completed   0          98s    10.42.0.9      ip-172-31-30-25   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-dv49r            0/1     Completed   2          98s    10.42.0.3      ip-172-31-30-25   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-bjvvh    0/1     Completed   0          98s    10.42.0.2      ip-172-31-30-25   <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-30-25                         1/1     Running     0          114s   172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-30-25                1/1     Running     0          112s   172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-30-25                             1/1     Running     0          109s   172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-30-25                         1/1     Running     0          113s   172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/rke2-canal-smkcq                                       2/2     Running     0          88s    172.31.30.25   ip-172-31-30-25   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-5c9bfbbf59-lmkf2             1/1     Running     0          90s    10.42.0.5      ip-172-31-30-25   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-df5d778cd-gcnkd   1/1     Running     0          90s    10.42.0.10     ip-172-31-30-25   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-pmbk4                    1/1     Running     0          41s    10.42.0.12     ip-172-31-30-25   <none>           <none>
kube-system   pod/rke2-metrics-server-f596c7cdb-6fhbp                    1/1     Running     0          50s    10.42.0.11     ip-172-31-30-25   <none>           <none>
kube-system   pod/rke2-snapshot-controller-5f665cf8b7-wtb6d              1/1     Running     0          39s    10.42.0.13     ip-172-31-30-25   <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-78f747449d-mjbft      1/1     Running     0          60s    10.42.0.6      ip-172-31-30-25   <none>           <none>


Airgap 2:
INFO[0002] Checking local image archives in /var/lib/rancher/rke2/agent/images for [REDACTED]/rancher/rke2-runtime:v1.26.14-rc3-rke2r1
WARN[0002] Failed to load runtime image [REDACTED]/rancher/rke2-runtime:v1.26.14-rc3-rke2r1 from tarball: no local image available for [REDACTED]/rancher/rke2-runtime:v1.26.14-rc3-rke2r1: not found in any file in /var/lib/rancher/rke2/agent/images: image not found
INFO[0002] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0002] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.26.14-rc3-rke2r1
W0227 23:41:10.874682    2998 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0227 23:41:11.094236    2998 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"


NAME                    STATUS   ROLES                       AGE     VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-26-183   Ready    control-plane,etcd,master   3h22m   v1.26.14+rke2r1   172.31.26.183   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws   containerd://1.7.11-k3s2

NAMESPACE     NAME                                                        READY   STATUS      RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-26-183               1/1     Running     0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/etcd-ip-172-31-26-183                                   1/1     Running     0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/helm-install-rke2-canal-dklnp                           0/1     Completed   0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/helm-install-rke2-coredns-2vrn9                         0/1     Completed   0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-4mqww                   0/1     Completed   0          3h21m   10.42.0.5       ip-172-31-26-183   <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-9n2vv                  0/1     Completed   0          3h21m   10.42.0.8       ip-172-31-26-183   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-8v76x         0/1     Completed   0          3h21m   10.42.0.4       ip-172-31-26-183   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-htvfc             0/1     Completed   1          3h21m   10.42.0.6       ip-172-31-26-183   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-wk7bb     0/1     Completed   0          3h21m   10.42.0.7       ip-172-31-26-183   <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-26-183                         1/1     Running     0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-26-183                1/1     Running     0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-26-183                             1/1     Running     0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-26-183                         1/1     Running     0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/rke2-canal-fthzk                                        2/2     Running     0          3h21m   172.31.26.183   ip-172-31-26-183   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-7ff974d84b-p4n86              1/1     Running     0          3h21m   10.42.0.3       ip-172-31-26-183   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-745dbfb5d5-phn72   1/1     Running     0          3h21m   10.42.0.2       ip-172-31-26-183   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-sktc7                     1/1     Running     0          3h20m   10.42.0.13      ip-172-31-26-183   <none>           <none>
kube-system   pod/rke2-metrics-server-7cd7c6dd89-bn5dx                    1/1     Running     0          3h20m   10.42.0.10      ip-172-31-26-183   <none>           <none>
kube-system   pod/rke2-snapshot-controller-645c4f6794-js64g               1/1     Running     0          3h20m   10.42.0.12      ip-172-31-26-183   <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-86755b6549-xdtht       1/1     Running     0          3h20m   10.42.0.9       ip-172-31-26-183   <none>           <none>

Additional context / logs:

N/A

brandond self-assigned this Feb 27, 2024

brandond added this to the v1.26.14+rke2r1 milestone Feb 27, 2024

brandond mentioned this issue Feb 27, 2024

[release-1.26] Backport bugfixes for 2024-02 rc #5533

Merged

endawkins self-assigned this Feb 27, 2024

endawkins closed this as completed Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Release-1.26] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5524

[Release-1.26] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5524

brandond commented Feb 27, 2024

endawkins commented Feb 28, 2024 •

edited

Loading

[Release-1.26] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5524

[Release-1.26] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5524

Comments

brandond commented Feb 27, 2024

endawkins commented Feb 28, 2024 • edited Loading

Validated on 1.26 with 9351840 / 1.26

Environment Details

Testing Steps

Air-Gap Setup

Pull-Through Cache Configuration

Copying Files to Airgapped Instance [Do this for each pair of airgapped instances]

Certificates and RKE2 Setup

Registries

Observations:

endawkins commented Feb 28, 2024 •

edited

Loading

Validated on 1.26 with `9351840` / 1.26