[Release-1.28] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5522

brandond · 2024-02-27T00:44:49Z

Backport fix for Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion

Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5516

endawkins · 2024-02-28T03:45:59Z

Validated on 1.28 with `fe8538a` / 1.28

Environment Details

Infrastructure

Cloud
Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-19-58 4.15.0-1051-aws #53-Ubuntu SMP Wed Sep 18 13:35:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Cluster Configuration:

4 Servers:
2 Bastion Hosts
2 Airgapped Instances

Config.yaml:

config.yaml 1:
token: test
debug: true
write-kubeconfig-mode: 644
system-default-registry: [REDACTED]

config.yaml 2:
token: test
debug: true
write-kubeconfig-mode: 644
system-default-registry: [REDACTED]

Additional files

registries.yaml
- airgap 1 [registry prefix] w/ uri schema
---
mirrors:
  docker.io:
    endpoint:
            - "https://[REDACTED]"
  [REDACTED]:
    endpoint:
            - "https://[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

airgap 1 [registry prefix] w/o uri schema

---
mirrors:
  docker.io:
    endpoint:
            - "[REDACTED]"
  [REDACTED]:
    endpoint:
            - "[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

registries.yaml w/o schema
- airgap 2 [no registry prefix]

---
mirrors:
  docker.io:
    endpoint:
      - "[REDACTED]"
  [REDACTED]:
    endpoint:
      - "[REDACTED]"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

Testing Steps

Note:

Bastion Instance 1 -> Airgap Instance 1
Bastion Instance 2 -> Airgap Instance 2

Air-Gap Setup

Launch two bastion instances from AWS
Launch two airgapped instances from AWS -- disable auto-assign public IP
ssh into the bastion nodes
Copy .pem file to bastion instances
scp -i "<path_to_pem_file>" <path_to_pem_file> username@<PUBLIC_IP>:~
ssh into airgapped instances

Pull-Through Cache Configuration

Add certificates:
mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt -subj "/C=US/ST=AZ/O=Rancher QA/CN=[REDACTED]" -addext "subjectAltName = DNS:[REDACTED]"
mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt -subj "/C=US/ST=AZ/O=Rancher QA/CN=[REDACTED]" -addext "subjectAltName = DNS:[REDACTED]"
Bastion 1:
sudo docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -e REGISTRY_HTTP_PREFIX=/testing/ -p 443:443 registry:2.7.1
Bastion 2:
sudo docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -p 443:443 registry:2.7.1
To view the containers running: docker ps -a

Copying Files to Airgapped Instance [Do this for each pair of airgapped instances]

Obtain the rke2 binary, rename it if desired (make sure its amd64): wget -O <NAME_THE_BINARY_FILE> https://github.com/rancher/rke2/releases/download/<VERSION>/<FILENAME>
ssh into airgapped instance
ssh -i <file_name>.pem username@<AIRGAP_IP>
close connection to airgapped instanceexit
copy rke2 binary and certificates to airgapped instance:

scp -i <file_name>.pem <rke2_binary_file> username@<AIRGAP_IP>:~
scp -i <file_name>.pem certs/* username@<AIRGAP_IP>:

ssh to airgapped instance

Certificates and RKE2 Setup

Update Certificates:
sudo cp domain.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
sudo vi config.yaml (there will be a config.yaml in both airgapped instances - a total of 2)
sudo vi registries (there will be a registries.yaml in both airgapped instances - a total of 2)
sudo mkdir -p /etc/rancher/rke2/ && sudo cp config.yaml /etc/rancher/rke2/ && cat /etc/rancher/rke2/config.yaml && sudo cp registries.yaml /etc/rancher/rke2/ && sudo cat /etc/rancher/rke2/registries.yaml
** Making RKE2 Binaries Executable **
chmod +x <RKE2_BINARY>
sudo mv <RKE2_BINARY> /usr/local/bin/rke2
Check version: rke2 --version
Open two new terminals and ssh into airgap instance 1 and 2

in those two terminals run the following command: sudo rke2 server

source .bashrc
kga
search for "?ns"

Replication Results:

rke2 version used for replication:

rke2 --version
rke2 version v1.26.13+rke2r1 (637e8a38334f603b60650b30547252a5c461fa0d)
go version go1.20.13 X:boringcrypto

Registries

registries.yaml
- airgap 1 [registry prefix] w/ uri schema
---
mirrors:
  docker.io:
    endpoint:
            - "https://[REDACTED]"
  [REDACTED]:
    endpoint:
            - "https://[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

airgap 1 [registry prefix] w/o uri schema

---
mirrors:
  docker.io:
    endpoint:
            - "[REDACTED]"
  [REDACTED]:
    endpoint:
            - "[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

registries.yaml w/o schema
- airgap 2 [no registry prefix]

---
mirrors:
  docker.io:
    endpoint:
      - "[REDACTED]"
  [REDACTED]:
    endpoint:
      - "[REDACTED]"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

Observations:

prefix + no uri schema:

Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/v2/?ns=[REDACTED]
W0227 22:57:05.649740    2954 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
WARN[0002] Failed to get image from endpoint: GET https://[REDACTED]/v2/?ns=REDACTED: unexpected status code 404 Not Found: 404 page not found
FATA[0002] failed to get runtime image [REDACTED]/rancher/rke2-runtime:v1.26.13-rke2r1: all endpoints failed: GET https://[REDACTED]/v2/?ns=REDACTED: unexpected status code 404 Not Found: 404 page not found

prefix + uri schema:

INFO[0002] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0002] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.26.13-rke2r1
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/testing/v2/?ns=[REDACTED]
W0227 22:58:16.196629    2966 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/v1.26.13-rke2r1 => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/v1.26.13-rke2r1?ns=[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED]?ns=[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED]?ns=[REDACTED]

Validation Results:

rke2 version used for validation:

rke2 --version
rke2 version v1.28.7-rc3+rke2r1 (fe8538a165d25755050d2a0827dbf982f72611fa)
go version go1.21.7 X:boringcrypto

rke2 starts successfully with no ?ns=<registry_hostname> in the logs

Airgap 1:
INFO[0002] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0002] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.28.7-rc3-rke2r1
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/testing/v2/
W0228 03:38:08.442570    2944 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/v1.28.7-rc3-rke2r1 => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/v1.28.7-rc3-rke2r1

NAME                    STATUS   ROLES                       AGE     VERSION          INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-31-244   Ready    control-plane,etcd,master   3m56s   v1.28.7+rke2r1   172.31.31.244   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws   containerd://1.7.11-k3s2

NAMESPACE     NAME                                                       READY   STATUS      RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-31-244              1/1     Running     0          3m45s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/etcd-ip-172-31-31-244                                  1/1     Running     0          3m34s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/helm-install-rke2-canal-t2xkt                          0/1     Completed   0          3m39s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/helm-install-rke2-coredns-rbd8w                        0/1     Completed   0          3m39s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-krtmp                  0/1     Completed   0          3m39s   10.42.0.8       ip-172-31-31-244   <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-lqmrj                 0/1     Completed   0          3m39s   10.42.0.7       ip-172-31-31-244   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-dwnqd        0/1     Completed   0          3m39s   10.42.0.3       ip-172-31-31-244   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-j5vrf            0/1     Completed   0          3m39s   10.42.0.6       ip-172-31-31-244   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-d94qp    0/1     Completed   0          3m39s   10.42.0.5       ip-172-31-31-244   <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-31-244                        1/1     Running     0          3m43s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-31-244               1/1     Running     0          3m44s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-31-244                            1/1     Running     0          3m48s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-31-244                        1/1     Running     0          3m44s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/rke2-canal-774g5                                       2/2     Running     0          3m29s   172.31.31.244   ip-172-31-31-244   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-677dbc9667-p5np5             1/1     Running     0          3m30s   10.42.0.4       ip-172-31-31-244   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-6b4d47b94-85mpj   1/1     Running     0          3m30s   10.42.0.2       ip-172-31-31-244   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-wlgcc                    1/1     Running     0          2m36s   10.42.0.13      ip-172-31-31-244   <none>           <none>
kube-system   pod/rke2-metrics-server-6b48d4997b-7gmlk                   1/1     Running     0          2m47s   10.42.0.11      ip-172-31-31-244   <none>           <none>
kube-system   pod/rke2-snapshot-controller-5769d9ff85-vbqx2              1/1     Running     0          2m48s   10.42.0.10      ip-172-31-31-244   <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-7c7764cf48-tnd9d      1/1     Running     0          2m51s   10.42.0.9       ip-172-31-31-244   <none>           <none>

Airgap 2:
INFO[0002] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0002] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.28.7-rc3-rke2r1
W0228 03:38:31.008442    2981 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"

NAME                    STATUS   ROLES                       AGE   VERSION          INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-26-131   Ready    control-plane,etcd,master   4m    v1.28.7+rke2r1   172.31.26.131   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws   containerd://1.7.11-k3s2

NAMESPACE     NAME                                                       READY   STATUS      RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-26-131              1/1     Running     0          3m35s   172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/etcd-ip-172-31-26-131                                  1/1     Running     0          3m6s    172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/helm-install-rke2-canal-8r7t6                          0/1     Completed   0          3m41s   172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/helm-install-rke2-coredns-bvmd2                        0/1     Completed   0          3m41s   172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-82prq                  0/1     Completed   0          3m41s   10.42.0.5       ip-172-31-26-131   <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-g8rgw                 0/1     Completed   0          3m41s   10.42.0.2       ip-172-31-26-131   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-zpshf        0/1     Completed   0          3m41s   10.42.0.4       ip-172-31-26-131   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-t5d28            0/1     Completed   0          3m41s   10.42.0.7       ip-172-31-26-131   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-rdx5j    0/1     Completed   0          3m41s   10.42.0.3       ip-172-31-26-131   <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-26-131                        1/1     Running     0          3m8s    172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-26-131               1/1     Running     0          3m37s   172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-26-131                            1/1     Running     0          3m30s   172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-26-131                        1/1     Running     0          3m37s   172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/rke2-canal-j4gxl                                       2/2     Running     0          3m30s   172.31.26.131   ip-172-31-26-131   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-588df87dcb-4t4j5             1/1     Running     0          3m31s   10.42.0.8       ip-172-31-26-131   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-9d6556995-plmqn   1/1     Running     0          3m31s   10.42.0.6       ip-172-31-26-131   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-tbwgj                    1/1     Running     0          2m35s   10.42.0.13      ip-172-31-26-131   <none>           <none>
kube-system   pod/rke2-metrics-server-78845947d9-42cw9                   1/1     Running     0          2m52s   10.42.0.9       ip-172-31-26-131   <none>           <none>
kube-system   pod/rke2-snapshot-controller-b7cb6fd4b-p55vk               1/1     Running     0          2m47s   10.42.0.11      ip-172-31-26-131   <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-776f84575c-fq8gg      1/1     Running     0          2m50s   10.42.0.10      ip-172-31-26-131   <none>           <none>

Additional context / logs:

N/A

brandond self-assigned this Feb 27, 2024

brandond added this to the v1.28.7+rke2r1 milestone Feb 27, 2024

brandond mentioned this issue Feb 27, 2024

[release-1.28] Backport bugfixes for 2024-02 rc #5531

Merged

endawkins self-assigned this Feb 27, 2024

endawkins closed this as completed Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Release-1.28] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5522

[Release-1.28] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5522

brandond commented Feb 27, 2024

endawkins commented Feb 28, 2024 •

edited

Loading

[Release-1.28] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5522

[Release-1.28] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5522

Comments

brandond commented Feb 27, 2024

endawkins commented Feb 28, 2024 • edited Loading

Validated on 1.28 with fe8538a / 1.28

Environment Details

Testing Steps

Air-Gap Setup

Pull-Through Cache Configuration

Copying Files to Airgapped Instance [Do this for each pair of airgapped instances]

Certificates and RKE2 Setup

Registries

Observations:

endawkins commented Feb 28, 2024 •

edited

Loading

Validated on 1.28 with `fe8538a` / 1.28