[Release-1.28] - opentelemetry can't be enabled with rke2-ingress-nginx

[dereknola]

Backport fix for opentelemetry can't be enabled with rke2-ingress-nginx

• opentelemetry can't be enabled with rke2-ingress-nginx #5539

[fmoral2]

Validated on Version:

-$   rke2 version v1.28.10+dev.95783fce (95783fcefeea0105a10e6c18f86d714d3f075d54)


 

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
ubuntu
AMD

Cluster Configuration:
-3 node server

Steps to validate the fix

1. Install rke2
2. Install Helm
3. Install and add https://github.com/esigo/nginx-example
4. Validate nginx-ingress-controller chart versions
5. Validate that can create pod for open-telemetry
6. Validate nodes and pods

Followed steps:
#6022 (comment)

Reproduction Issue:

 rke2 version v1.30.1+dev.d2cae594 (d2cae59491bdab3614a017e4496c5b678b6aa6d7)



 kubectl get node -o yaml | grep 'nginx-ingress-controller' -A1
      - docker.io/rancher/nginx-ingress-controller@sha256:54bf3c4ff63758d44fd2dd886fec40783fa2005165e3fea2999f69f40d2a7210
      - docker.io/rancher/nginx-ingress-controller:nginx-1.9.6-hardened1
      sizeBytes: 325093593
--
      - docker.io/rancher/nginx-ingress-controller@sha256:54bf3c4ff63758d44fd2dd886fec40783fa2005165e3fea2999f69f40d2a7210
      - docker.io/rancher/nginx-ingress-controller:nginx-1.9.6-hardened1
      sizeBytes: 325093593
--
      - docker.io/rancher/nginx-ingress-controller@sha256:54bf3c4ff63758d44fd2dd886fec40783fa2005165e3fea2999f69f40d2a7210
      - docker.io/rancher/nginx-ingress-controller:nginx-1.9.6-hardened1
      sizeBytes: 325093593
ubuntu@ip-172-31-0-123:~/nginx-example$ make deploy-app
kubectl apply -f micro_app/namespace.yaml
namespace/myapp created
kubectl apply -f micro_app/microapp-a.yaml
deployment.apps/microapp created
service/microapp-service created
Error from server (BadRequest): error when creating "micro_app/microapp-a.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
-------------------------------------------------------------------------------
Error: exit status 1
2024/06/10 19:28:39 [emerg] 161#161: dlopen() "/modules_mount/etc/nginx/modules/otel/otel_ngx_module.so" failed (/modules_mount/etc/nginx/modules/otel/otel_ngx_module.so: cannot open shared object file: No such file or directory) in /tmp/nginx/nginx-cfg1731248817:7
nginx: [emerg] dlopen() "/modules_mount/etc/nginx/modules/otel/otel_ngx_module.so" failed (/modules_mount/etc/nginx/modules/otel/otel_ngx_module.so: cannot open shared object file: No such file or directory) in /tmp/nginx/nginx-cfg1731248817:7
nginx: configuration file /tmp/nginx/nginx-cfg1731248817 test failed

-------------------------------------------------------------------------------
make: [Makefile:9: deploy-app] Error 1 (ignored)
kubectl apply -f micro_app/microapp-b.yaml
deployment.apps/microapp-b created
service/microapp-service-b created
bash ./wait.sh _wait "pod -l app=demo-microapp -n myapp"
^CWaiting for 'pod -l app=demo-microapp -n myapp'
^Cmake: [Makefile:11: deploy-app] Interrupt (ignored)

Validation Results:

 
 
 
  
make deploy-app
kubectl apply -f micro_app/namespace.yaml
namespace/myapp unchanged
kubectl apply -f micro_app/microapp-a.yaml
deployment.apps/microapp unchanged
service/microapp-service unchanged
ingress.networking.k8s.io/microapp unchanged
kubectl apply -f micro_app/microapp-b.yaml
deployment.apps/microapp-b unchanged
service/microapp-service-b unchanged
bash ./wait.sh _wait "pod -l app=demo-microapp -n myapp"
Waiting for 'pod -l app=demo-microapp -n myapp'
Waiting for 'pod -l app=demo-microapp -n myapp'


curl -u admin:admin http://localhost:3000/api/health
{
  "commit": "83b9528bce85cf9371320f6d6e450916156da3f6",
  "database": "ok",
  "version": "11.0.0"


$ k get pods -A
NAMESPACE       NAME                                                                  READY   STATUS              RESTARTS   AGE
cert-manager    cert-manager-7dff8ccdd7-5grnt                                         1/1     Running             0          3h17m
cert-manager    cert-manager-cainjector-5bb8444c78-x4j7m                              1/1     Running             0          3h17m
cert-manager    cert-manager-webhook-9594fd5f-bgmvl                                   1/1     Running             0          3h17m
kube-system     cloud-controller-manager-.us-east-2.compute.internal   1/1     Running             0          3h38m
kube-system     cloud-controller-manager-ip-.us-east-2.compute.internal    1/1     Running             0          3h36m
kube-system     cloud-controller-manager-ip-.us-east-2.compute.internal    1/1     Running             0          3h35m
kube-system     etcd-.us-east-2.compute.internal                       1/1     Running             0          3h38m
kube-system     etcd-ip-.us-east-2.compute.internal                        1/1     Running             0          3h35m
kube-system     etcd-ip-.us-east-2.compute.internal                        1/1     Running             0          3h35m
kube-system     helm-install-rke2-canal-6cdc9                                         0/1     Completed           0          3h38m
kube-system     helm-install-rke2-coredns-qn4bm                                       0/1     Completed           0          3h38m
kube-system     helm-install-rke2-ingress-nginx-twcnw                                 0/1     Completed           0          25m
kube-system     helm-install-rke2-metrics-server-b49hz                                0/1     Completed           0          3h38m
kube-system     helm-install-rke2-snapshot-controller-crd-x6m5r                       0/1     Completed           0          3h38m
kube-system     helm-install-rke2-snapshot-controller-ldpz6                           0/1     Completed           1          3h38m
kube-system     helm-install-rke2-snapshot-validation-webhook-qc8hm                   0/1     Completed           0          3h38m
kube-system     kube-apiserver-.us-east-2.compute.internal             1/1     Running             0          3h38m
kube-system     kube-apiserver-ip-.us-east-2.compute.internal              1/1     Running             0          3h35m
kube-system     kube-apiserver-ip-.us-east-2.compute.internal              1/1     Running             0          3h35m
kube-system     kube-controller-manager-.us-east-2.compute.internal    1/1     Running             0          3h38m
kube-system     kube-controller-manager-ip-.us-east-2.compute.internal     1/1     Running             0          3h35m
kube-system     kube-controller-manager-ip-.us-east-2.compute.internal     1/1     Running             0          3h35m
kube-system     kube-proxy-.us-east-2.compute.internal                 1/1     Running             0          3h38m
kube-system     kube-proxy-ip-.us-east-2.compute.internal                  1/1     Running             0          3h35m
kube-system     kube-proxy-ip-.us-east-2.compute.internal                  1/1     Running             0          3h35m
kube-system     kube-scheduler-.us-east-2.compute.internal             1/1     Running             0          3h38m
kube-system     kube-scheduler-ip-.us-east-2.compute.internal              1/1     Running             0          3h35m
kube-system     kube-scheduler-ip-.us-east-2.compute.internal              1/1     Running             0          3h35m
kube-system     rke2-canal-2kxrq                                                      2/2     Running             0          3h36m
kube-system     rke2-canal-2x26t                                                      2/2     Running             0          3h38m
kube-system     rke2-canal-66lds                                                      2/2     Running             0          3h35m
kube-system     rke2-coredns-rke2-coredns-64dcf4f58b-s9p7r                            1/1     Running             0          3h38m
kube-system     rke2-coredns-rke2-coredns-64dcf4f58b-w5b6v                            1/1     Running             0          3h36m
kube-system     rke2-coredns-rke2-coredns-autoscaler-5749cd7b8b-xgpgs                 1/1     Running             0          3h38m
kube-system     rke2-ingress-nginx-controller-8wqzq                                   1/1     Running             0          24m
kube-system     rke2-ingress-nginx-controller-fj6qg                                   1/1     Running             0          23m
kube-system     rke2-ingress-nginx-controller-g4lh8                                   1/1     Running             0          24m
kube-system     rke2-metrics-server-868fc8795f-gl52m                                  1/1     Running             0          3h37m
kube-system     rke2-snapshot-controller-7dcf5d5b46-n5t86                             1/1     Running             0          3h37m
kube-system     rke2-snapshot-validation-webhook-bf7bbd6fc-gbtp7                      1/1     Running             0          3h37m
myapp           microapp-b-b6c57b867-njkzx                                            1/1     Running             0          14m
observability   grafana-855948b67c-qfcgn                                              1/1     Running             0          3h16m
observability   tempo-0                                                               1/1     Running             0          3h16m
otel            otel-coll-collector-6bb47d7675-w5rsr                                  1/1     Running             0          3h15m
otel            otel-collector-operator-opentelemetry-operator-6fc7fdb48f-5pjgs       2/2     Running             0          3h16m