caa-provisioner-cli -action=install has odd dependency on cmctl

[gcoon151]

Describe the bug

I'll look at finding/fixing also but I wanted to capture a note about something odd. I'm on Mac OS Sonoma 14.6.1 following recently updated instructions to deploy on OCP in IBM cloud. It got to here and died:

validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
# Wait for service to be up
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager condition met
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager-webhook --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager-webhook condition met
# Wait for few seconds for the cert-manager API to be ready
# otherwise you'll hit the error "x509: certificate signed by unknown authority"
# Best is to use cmctl - https://cert-manager.io/docs/installation/kubectl/#2-optional-wait-for-cert-manager-webhook-to-be-ready
./cmctl check api --wait=2m
bash: ./cmctl: cannot execute binary file
make: *** [deploy-cert-manager] Error 126 
FATA[0061] exit status 2   

I hadn't installed or even realized at that pointed I would need cmctl. brew install cmctl && cmctl check api --wait=2m
The cert-manager API is ready
Assuming it had completed like the output said, I rerun ./caa-provisioner-cli -action=install
where it again fails same way

validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
# Wait for service to be up
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager condition met
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager-webhook --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager-webhook condition met
# Wait for few seconds for the cert-manager API to be ready
# otherwise you'll hit the error "x509: certificate signed by unknown authority"
# Best is to use cmctl - https://cert-manager.io/docs/installation/kubectl/#2-optional-wait-for-cert-manager-webhook-to-be-ready
./cmctl check api --wait=2m
bash: ./cmctl: cannot execute binary file
make: *** [deploy-cert-manager] Error 126 
FATA[0061] exit status 2    

Is it trying to run cmctl from literally whatever directory I'm in? I'll help chase this down when I have a moment. I just didn't want to forget to open an issue about this.

How to reproduce

I was going through these instructions https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/ibmcloud/ROKS_SETUP.md and got all the way to
( ~/gits/cloud-api-adaptor/src/cloud-api-adaptor/test/tools/) $ ./caa-provisioner-cli -action=install

CoCo version information

main from this repo.

What TEE are you seeing the problem on

Tdx

Failing command and relevant log output

( ~/gits/cloud-api-adaptor/src/cloud-api-adaptor/test/tools/) $  ./caa-provisioner-cli -action=install

...

validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
# Wait for service to be up
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager condition met
kubectl wait --timeout=90s -n cert-manager endpoints/cert-manager-webhook --for=jsonpath='{.subsets[0].addresses[0].ip}'
endpoints/cert-manager-webhook condition met
# Wait for few seconds for the cert-manager API to be ready
# otherwise you'll hit the error "x509: certificate signed by unknown authority"
# Best is to use cmctl - https://cert-manager.io/docs/installation/kubectl/#2-optional-wait-for-cert-manager-webhook-to-be-ready
./cmctl check api --wait=2m
bash: ./cmctl: cannot execute binary file
make: *** [deploy-cert-manager] Error 126 
FATA[0061] exit status 2

[stevenhorsman]

I think this is because the curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_linux_${ARCH} assumes you are running linux, not mac, I'm guessing you might need darwin_arm64 or something instead?

[gcoon151]

I think you might be right. I think Frank ran into similar and extended arch to support what the target cluster would have as arch. Perhaps we need to have CLUSTER_ARCH and CLIENT_ARCH. I'll look more in there on mine too and see if I can confirm this and come up with a fix PR for you.

[gcoon151]

I actually just noticed this which might help

make BUILTIN_CLOUD_PROVIDERS="ibmcloud" all
GOOS=darwin GOARCH=arm64 CGO_ENABLED=0 go build -tags=ibmcloud -o caa-provisioner-cli ./provisioner-cli/

Seems like those vars could be reused for cmdctl?

[gcoon151]

I have another option in the works also of just examining the curl we're using to get cmctl and just match that platform. Let me test out both and I'll send over a PR on this to fix this. Something like

curl --version | awk 'NR==1 && match($0, /\(.*\)/) {
    platform=substr($0, RSTART+1, RLENGTH-2); 
    split(platform, a, "-"); 
    if (length(a) >= 3) { 
        os=a[3]; arch=a[1]; 
        if (arch == "x86_64") arch="amd64"; 
        else if (arch ~ /^(arm64|aarch64)$/) arch="arm64"; 
        if (os ~ /^(linux|darwin|windows)$/) print os "_" arch; 
        else print "unsupported"; 
    } else print "unsupported";
}'

This output generates based on the curl string something like linux_amd64 which can be assigned to a variable CLIENT_ARCH? and then get the right file. Just saving the thought here for my own notes. I'll PR this up as a suggestion.

[stevenhorsman]

Can you do just do something like:

curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_$(uname | tr '[:upper:]' '[:lower:]')_${ARCH}

to select the correct os for the download? It seems much simpler than your curl approach?

[gcoon151]

ARCH has to be defined by something and right now I think we're only letting you define the target platform for the podvm image which doesn't match the orchestrator/laptop that's instantiating caa. The core problem is my laptop is ARM so I need cmctl for ARM but I'm deploying to amd64 which is what I pass the caa-provisioner-cli

[stevenhorsman]

ARCH is defined here:

cloud-api-adaptor/src/webhook/Makefile

Line 1 in 464f734

ARCH ?= $(subst x86_64,amd64,$(shell uname -m))

and is using uname -m, so we match the machine it is running on?

[gcoon151]

I'm rebuilding the env today where I have some tdx hosts. When I get to that part I'll examine what my ARCH is set to and dig deeper on why it didn't get the right thing. I thought it was defined from the flag Frank added to specify arch.

[gcoon151]

I just noticed earlier, you said we grab curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_linux_${ARCH} (note "linux") when I look here https://github.com/cert-manager/cmctl/releases/ the one I need is darwin_arm64. uname -m only gives you the arm64. I think the problem might be a little further left in the URL where we presume linux ... which you can also get from the curl if we decide to do that.

Golang defines OS and ARCH. Seems this Makefile only defines ARCH and assumes linux.

[gcoon151]

Oh I should add one reason I went with the curl specifics to identify OS/ARCH is I tested what the uname output would look like if someone ran this code in Ubuntu on WSL on Windows. Obviously not supported today but that's what prompted me to consider a wider method of defining these.

[stevenhorsman]

I just noticed earlier, you said we grab curl -fsSL -o cmctl https://github.com/cert-manager/cmctl/releases/latest/download/cmctl_linux_${ARCH} (note "linux") when I look here https://github.com/cert-manager/cmctl/releases/ the one I need is darwin_arm64. uname -m only gives you the arm64. I think the problem might be a little further left in the URL where we presume linux ... which you can also get from the curl if we decide to do that.

Yes, this is why I have $(uname | tr '[:upper:]' '[:lower:]') in my suggested fix:

$ uname | tr '[:upper:]' '[:lower:]'
darwin

[gcoon151]

I missed that and confirmed on ubuntu on wsl which sent me down this path that'd return linux also so that would be a good approach.