Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.27] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5523

Closed
brandond opened this issue Feb 27, 2024 · 1 comment
Closed

[Release-1.27] - Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion #5523

brandond opened this issue Feb 27, 2024 · 1 comment
Assignees
@brandond @endawkins
Milestone
v1.27.11+rke2r1

Comments

@brandond
Copy link
Member

Backport fix for Add support for bare hostname as endpoint, fix unnecessary namespace param inclusion
@brandond brandond self-assigned this Feb 27, 2024
@brandond brandond added this to the v1.27.11+rke2r1 milestone Feb 27, 2024
@brandond brandond mentioned this issue Feb 27, 2024
Merged
@endawkins endawkins self-assigned this Feb 27, 2024
@endawkins
Copy link

Validated on 1.27 with 6665618 / 1.27

Environment Details

Infrastructure

  • Cloud
  • Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-19-58 4.15.0-1051-aws #53-Ubuntu SMP Wed Sep 18 13:35:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Cluster Configuration:

4 Servers:
2 Bastion Hosts
2 Airgapped Instances

Config.yaml:

config.yaml 1:
token: test
debug: true
write-kubeconfig-mode: 644
system-default-registry: [REDACTED]

config.yaml 2:
token: test
debug: true
write-kubeconfig-mode: 644
system-default-registry: [REDACTED]

Additional files

registries.yaml
- airgap 1 [registry prefix] w/ uri schema
---
mirrors:
  docker.io:
    endpoint:
            - "https://[REDACTED]"
  [REDACTED]:
    endpoint:
            - "https://[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

airgap 1 [registry prefix] w/o uri schema

---
mirrors:
  docker.io:
    endpoint:
            - "[REDACTED]"
  [REDACTED]:
    endpoint:
            - "[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

registries.yaml w/o schema
- airgap 2 [no registry prefix]

---
mirrors:
  docker.io:
    endpoint:
      - "[REDACTED]"
  [REDACTED]:
    endpoint:
      - "[REDACTED]"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

Testing Steps

Note:

Bastion Instance 1 -> Airgap Instance 1
Bastion Instance 2 -> Airgap Instance 2

Air-Gap Setup

  1. Launch two bastion instances from AWS
  2. Launch two airgapped instances from AWS -- disable auto-assign public IP
  3. ssh into the bastion nodes
  4. Copy .pem file to bastion instances
    scp -i "<path_to_pem_file>" <path_to_pem_file> username@<PUBLIC_IP>:~
  5. ssh into airgapped instances

Pull-Through Cache Configuration

  1. Add certificates:
    mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt -subj "/C=US/ST=AZ/O=Rancher QA/CN=[REDACTED]" -addext "subjectAltName = DNS:[REDACTED]"
    mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt -subj "/C=US/ST=AZ/O=Rancher QA/CN=[REDACTED]" -addext "subjectAltName = DNS:[REDACTED]"
  2. Bastion 1:
    sudo docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -e REGISTRY_HTTP_PREFIX=/testing/ -p 443:443 registry:2.7.1
    Bastion 2:
    sudo docker run -d --restart=always --name registry -v "$(pwd)"/certs:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -p 443:443 registry:2.7.1
  3. To view the containers running: docker ps -a

Copying Files to Airgapped Instance [Do this for each pair of airgapped instances]

  1. Obtain the rke2 binary, rename it if desired (make sure its amd64): wget -O <NAME_THE_BINARY_FILE> https://github.com/rancher/rke2/releases/download/<VERSION>/<FILENAME>
  2. ssh into airgapped instance
    ssh -i <file_name>.pem username@<AIRGAP_IP>
  3. close connection to airgapped instanceexit
  4. copy rke2 binary and certificates to airgapped instance:
scp -i <file_name>.pem <rke2_binary_file> username@<AIRGAP_IP>:~
scp -i <file_name>.pem certs/* username@<AIRGAP_IP>:
  1. ssh to airgapped instance

Certificates and RKE2 Setup

  1. Update Certificates:
    sudo cp domain.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
  2. sudo vi config.yaml (there will be a config.yaml in both airgapped instances - a total of 2)
  3. sudo vi registries (there will be a registries.yaml in both airgapped instances - a total of 2)
  4. sudo mkdir -p /etc/rancher/rke2/ && sudo cp config.yaml /etc/rancher/rke2/ && cat /etc/rancher/rke2/config.yaml && sudo cp registries.yaml /etc/rancher/rke2/ && sudo cat /etc/rancher/rke2/registries.yaml
    ** Making RKE2 Binaries Executable **
  5. chmod +x <RKE2_BINARY>
  6. sudo mv <RKE2_BINARY> /usr/local/bin/rke2
  7. Check version: rke2 --version
  8. Open two new terminals and ssh into airgap instance 1 and 2
  • in those two terminals run the following command: sudo rke2 server
  1. source .bashrc
  2. kga
  3. search for "?ns"

Replication Results:

  • rke2 version used for replication:
rke2 --version
rke2 version v1.26.13+rke2r1 (637e8a38334f603b60650b30547252a5c461fa0d)
go version go1.20.13 X:boringcrypto

Registries

registries.yaml
- airgap 1 [registry prefix] w/ uri schema
---
mirrors:
  docker.io:
    endpoint:
            - "https://[REDACTED]"
  [REDACTED]:
    endpoint:
            - "https://[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

airgap 1 [registry prefix] w/o uri schema

---
mirrors:
  docker.io:
    endpoint:
            - "[REDACTED]"
  [REDACTED]:
    endpoint:
            - "[REDACTED]/testing/v2/"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

registries.yaml w/o schema
- airgap 2 [no registry prefix]

---
mirrors:
  docker.io:
    endpoint:
      - "[REDACTED]"
  [REDACTED]:
    endpoint:
      - "[REDACTED]"
configs:
  "[REDACTED]":
    tls:
      cert_file: /home/ubuntu/domain.crt
      key_file: /home/ubuntu/domain.key

Observations:

prefix + no uri schema:

Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/v2/?ns=[REDACTED]
W0227 22:57:05.649740    2954 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
WARN[0002] Failed to get image from endpoint: GET https://[REDACTED]/v2/?ns=REDACTED: unexpected status code 404 Not Found: 404 page not found
FATA[0002] failed to get runtime image [REDACTED]/rancher/rke2-runtime:v1.26.13-rke2r1: all endpoints failed: GET https://[REDACTED]/v2/?ns=REDACTED: unexpected status code 404 Not Found: 404 page not found

prefix + uri schema:

INFO[0002] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0002] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0002] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.26.13-rke2r1
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/testing/v2/?ns=[REDACTED]
W0227 22:58:16.196629    2966 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
DEBU[0002] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/v1.26.13-rke2r1 => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/v1.26.13-rke2r1?ns=[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED]?ns=[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED]?ns=[REDACTED]

Validation Results:

  • rke2 version used for validation:
rke2 --version
rke2 version v1.27.11-rc3+rke2r1 (6665618680112568f79b1f5992aecf4655e3cf8b)
go version go1.21.7 X:boringcrypto

rke2 starts successfully with no ?ns=<registry_hostname> in the logs

Airgap 1:
INFO[0003] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0003] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0003] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.27.11-rc3-rke2r1
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/ => https://[REDACTED]/testing/v2/
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/v1.27.11-rc3-rke2r1 => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/v1.27.11-rc3-rke2r1
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/manifests/sha256:[REDACTED]
DEBU[0003] Registry endpoint URL modified: https://[REDACTED]/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED] => https://[REDACTED]/testing/v2/rancher/rke2-runtime/blobs/sha256:[REDACTED]
INFO[0003] Creating directory /var/lib/rancher/rke2/data/v1.27.11-rc3-rke2r1-ed4327473f2e/bin

NAME                    STATUS   ROLES                       AGE    VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-16-197   Ready    control-plane,etcd,master   3m9s   v1.27.11+rke2r1   172.31.16.197   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws   containerd://1.7.11-k3s2

NAMESPACE     NAME                                                        READY   STATUS      RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-16-197               1/1     Running     0          2m8s    172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/etcd-ip-172-31-16-197                                   1/1     Running     0          117s    172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/helm-install-rke2-canal-5p2fr                           0/1     Completed   0          2m52s   172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/helm-install-rke2-coredns-56zd7                         0/1     Completed   0          2m52s   172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-2qjh2                   0/1     Completed   0          2m52s   10.42.0.6       ip-172-31-16-197   <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-wdwjd                  0/1     Completed   0          2m52s   10.42.0.2       ip-172-31-16-197   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-9hz5v         0/1     Completed   0          2m51s   10.42.0.4       ip-172-31-16-197   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-v24zt             0/1     Completed   0          2m52s   10.42.0.7       ip-172-31-16-197   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-m2dwz     0/1     Completed   0          2m51s   10.42.0.3       ip-172-31-16-197   <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-16-197                         1/1     Running     0          2m14s   172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-16-197                1/1     Running     0          2m12s   172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-16-197                             1/1     Running     0          2m1s    172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-16-197                         1/1     Running     0          2m16s   172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/rke2-canal-5tsz6                                        2/2     Running     0          2m18s   172.31.16.197   ip-172-31-16-197   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-5d5676d6d5-q7ppg              1/1     Running     0          2m19s   10.42.0.8       ip-172-31-16-197   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-6bfb9d7844-j9rpk   1/1     Running     0          2m19s   10.42.0.5       ip-172-31-16-197   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-sqcws                     1/1     Running     0          76s     10.42.0.13      ip-172-31-16-197   <none>           <none>
kube-system   pod/rke2-metrics-server-66cdd8567d-s8mw2                    1/1     Running     0          95s     10.42.0.9       ip-172-31-16-197   <none>           <none>
kube-system   pod/rke2-snapshot-controller-55b6d6974c-vtbx2               1/1     Running     0          90s     10.42.0.11      ip-172-31-16-197   <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-7fd44bc47d-4bvrc       1/1     Running     0          91s     10.42.0.10      ip-172-31-16-197   <none>           <none>

Airgap 2:
INFO[0003] Using private registry config file at /etc/rancher/rke2/registries.yaml
DEBU[0003] Kubelet image credential provider bin directory check failed: stat /var/lib/rancher/credentialprovider/bin: no such file or directory
INFO[0003] Pulling runtime image [REDACTED]/rancher/rke2-runtime:v1.27.11-rc3-rke2r1
INFO[0003] Creating directory /var/lib/rancher/rke2/data/v1.27.11-rc3-rke2r1-ed4327473f2e/bin
INFO[0003] Extracting file bin/containerd to /var/lib/rancher/rke2/data/v1.27.11-rc3-rke2r1-ed4327473f2e/bin/containerd
INFO[0004] Extracting file bin/containerd-shim to /var/lib/rancher/rke2/data/v1.27.11-rc3-rke2r1-ed4327473f2e/bin/containerd-shim
INFO[0004] Extracting file bin/containerd-shim-runc-v1 to /var/lib/rancher/rke2/data/v1.27.11-rc3-rke2r1-ed4327473f2e/bin/containerd-shim-runc-v1
INFO[0005] Extracting file bin/containerd-shim-runc-v2 to /var/lib/rancher/rke2/data/v1.27.11-rc3-rke2r1-ed4327473f2e/bin/containerd-shim-runc-v2
W0228 12:16:59.695008   15379 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"

NAME                   STATUS   ROLES                       AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-28-21   Ready    control-plane,etcd,master   3m23s   v1.27.11+rke2r1   172.31.28.21   <none>        Ubuntu 18.04.3 LTS   4.15.0-1051-aws   containerd://1.7.11-k3s2

NAMESPACE     NAME                                                        READY   STATUS      RESTARTS   AGE     IP             NODE              NOMINATED NODE   READINESS GATES
kube-system   pod/cloud-controller-manager-ip-172-31-28-21                1/1     Running     0          3m18s   172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/etcd-ip-172-31-28-21                                    1/1     Running     0          2m58s   172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/helm-install-rke2-canal-xx7cw                           0/1     Completed   0          3m3s    172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/helm-install-rke2-coredns-z9jp5                         0/1     Completed   0          3m3s    172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/helm-install-rke2-ingress-nginx-2n5br                   0/1     Completed   0          3m3s    10.42.0.7      ip-172-31-28-21   <none>           <none>
kube-system   pod/helm-install-rke2-metrics-server-blbww                  0/1     Completed   0          3m3s    10.42.0.5      ip-172-31-28-21   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-crd-4ssjj         0/1     Completed   0          3m3s    10.42.0.4      ip-172-31-28-21   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-controller-p42dt             0/1     Completed   0          3m3s    10.42.0.6      ip-172-31-28-21   <none>           <none>
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-ldt8l     0/1     Completed   0          3m3s    10.42.0.2      ip-172-31-28-21   <none>           <none>
kube-system   pod/kube-apiserver-ip-172-31-28-21                          1/1     Running     0          3m11s   172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/kube-controller-manager-ip-172-31-28-21                 1/1     Running     0          3m20s   172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/kube-proxy-ip-172-31-28-21                              1/1     Running     0          3m12s   172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/kube-scheduler-ip-172-31-28-21                          1/1     Running     0          3m20s   172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/rke2-canal-h4tg7                                        2/2     Running     0          2m53s   172.31.28.21   ip-172-31-28-21   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-6797fc8bd6-n4v7r              1/1     Running     0          2m54s   10.42.0.3      ip-172-31-28-21   <none>           <none>
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-5fc5d7f5b5-bp9vj   1/1     Running     0          2m54s   10.42.0.8      ip-172-31-28-21   <none>           <none>
kube-system   pod/rke2-ingress-nginx-controller-rzk66                     1/1     Running     0          118s    10.42.0.13     ip-172-31-28-21   <none>           <none>
kube-system   pod/rke2-metrics-server-5cf8d5766c-58jt5                    1/1     Running     0          2m10s   10.42.0.10     ip-172-31-28-21   <none>           <none>
kube-system   pod/rke2-snapshot-controller-86dfb877fb-9hg8v               1/1     Running     0          2m10s   10.42.0.11     ip-172-31-28-21   <none>           <none>
kube-system   pod/rke2-snapshot-validation-webhook-65676cbdd7-f7892       1/1     Running     0          2m16s   10.42.0.9      ip-172-31-28-21   <none>           <none>

Additional context / logs:

N/A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@endawkins endawkins

Labels
None yet
Projects
None yet
Milestone
v1.27.11+rke2r1
Development

No branches or pull requests
2 participants
@brandond @endawkins