ARM64 Support

[bryson]

I'd like to start testing support for my workloads on AWS's Graviton-based EC2 instances. I see that this isn't in the backlog yet; hopefully this Issue can get labeled so its progress can be monitored.

[bryson]

According to the folks at Rancher that I spoke with, the blocker on this issue is a Go lang issue: golang/go#39760

[brandond]

That is correct.

[bryson]

This issue is still important to me.

[simonflood]

I don't think this issue should have been closed, not sure if stale bot will now re-open it.

[brandond]

Sticking a label on it should keep it from going stale.

[belgaied2]

This comment suggests that sufficient documentation might avoid misunderstandings around FIPS certifications on ARM platforms and I agree with this.
RKE2 is becoming the mainstream datacenter distribution from Rancher, Government usecases are valuable but can be pitched correctly about non-support of ARM platforms.

I personally don't think FIPS certification of dependency libraries should be a blocker to ARM64 support on RKE2 or Harvester.

[codispatch]

Looks like The main blocker "Support BoringCrypto ARM64" for Go-lang is almost done and it is pending in go.dev/cl/423362 branch

[gizmotronic]

For what it's worth, go.dev/cl/423362 has now been merged.

[rajivml]

Am trying to setup k8's dev environment on Mac M1 and our installer is RHEL specific so I am using utm + Rocky Linux EL9 ISO image to spin up the environment but I just now realised that RKE2 doesn't support ARM64/aarch64, can we have this support added please

[javad-inlive]

Same issue trying to deploy in OCI. Running installation script throws an error [ERROR] unsupported architecture aarch64
Is there any news about aarch64 support?

[otterley]

Hi everyone,

On 2023-01-04 BoringCrypto was validated for FIPS-140-2 compliance on the arm64 CPU architecture (certificate 4407). So, if the RKE2 Golang-based components are built using the validated version of BoringSSL (https://boringssl.googlesource.com/boringssl/+/refs/tags/fips-20210429), Rancher should be able to have RKE2 be validated for FIPS-140-2 compliance on arm64.

[sdemura]

Hi, is there anything from an OSS contributor standpoint that I could do to help drive this?

[brandond]

Not really, no. I believe we need to line up a few things on the CI side, in addition to getting approval from a couple different teams (product/support/qa), before we can move forward on this.
cc @cwayne18

[mbehrisch]

Two quick questions here:
(1) any updates on this?
(2) I read on the frontpage of boringssl the following description "Although BoringSSL is an open source project, it is not intended for general use, as OpenSSL is. We don't recommend that third parties depend upon it. Doing so is likely to be frustrating because there are no guarantees of API or ABI stability." Does it make sense to rely the entire FIPS certification on such a lib?

[brandond]

We're not relying on it, golang is. If there are changes to the api or abi, the golang project maintainers will manage that when updating the goboring code base, which is in the mainline golang branches as of Go 1.19. Also, api stability guarantees don't really have anything to do with whether or not the actual crypto implementation is or can be certified. So yes, we believe it makes sense to rely upon.

[mbehrisch]

Thanks for the clarifications. I'm trying to learn her. These certification processes and particularly the transitive chains thereof are fully opaque to me. Thank you for taking the time to explain it.

[ganniterix]

Does this mean it's not going to drop for v1.26?

[rajivml]

based on the tagging, looks like it was put under Backlog again https://github.com/rancher/rke2/milestone/129

[mbehrisch]

According to my understanding, the only points for closing/finishing this issue are to update the installer and the documentation (modulus the internal politics on whether or not to support another platform). I don't hope the backlog tag de-facto kills this issue? I would love to hear where in the process this issue is stuck and whether or not an individual contributor can help.

[brandond]

• Add ARM64 Support image-build-base#32
• ARM64 Support image-build-kubernetes#43

[cwayne18]

This is being actively worked on, though to be clear there's a lot that needs to be defined in terms of support, but the goal is to have ARM64 builds in the near future, so this is very far from being dropped :)

[ghost]

This container has no arm64 image available and I can't figure out what repo has the CI for it:

• https://hub.docker.com/r/rancher/pause/tags/

These ones oddly have arm64 images, but seem to lack multi-arch support for the main tags. Not sure if this was intentional:

• https://hub.docker.com/r/rancher/hardened-multus-cni/tags/
• https://hub.docker.com/r/rancher/hardened-sriov-network-operator/tags/
• https://hub.docker.com/r/rancher/hardened-sriov-network-config-daemon/tags/
• https://hub.docker.com/r/rancher/hardened-sriov-network-device-plugin/tags/
• https://hub.docker.com/r/rancher/hardened-ib-sriov-cni/tags/
• https://hub.docker.com/r/rancher/hardened-sriov-network-resources-injector/tags/
• https://hub.docker.com/r/rancher/hardened-sriov-network-webhook/tags/

I noticed these images are referenced for the RKE2 build, but might not be relevant for arm64. Some of these aren't event relevant images afaik, like the cilium etcd operator is dead code based on the upstream repo:

• https://hub.docker.com/r/rancher/hardened-sriov-cni/tags/
• https://hub.docker.com/r/rancher/harvester-cloud-provider/tags/
• https://hub.docker.com/r/rancher/harvester-csi-driver/tags/
• https://hub.docker.com/r/rancher/mirrored-cilium-cilium-etcd-operator/tags/
• https://hub.docker.com/r/rancher/mirrored-cloud-provider-vsphere-cpi-release-manager/tags/
• https://hub.docker.com/r/rancher/mirrored-cloud-provider-vsphere-csi-release-driver/tags/
• https://hub.docker.com/r/rancher/mirrored-cloud-provider-vsphere-csi-release-syncer/tags/

I think once the pause container has an arm64 build available, changes can be made in rancher/rke2 for the build to work. The oddly tagged images aren't really a blocker, just inconvenient.

[briandowns]

The work for this is nearly complete. Thanks for this info.

[liyang516]

@briandowns When will the arm architecture be supported?

[briandowns]

It will be in this coming release (tomorrow?) but marked experimental and not official.

[liyang516]

@briandowns Hi，can you provide a experimental package for arm architecture,

[brandond]

@liyang516 an experimental package of what? The release artifacts for arm64 are on github now, what else are you looking for?

Note that we are only publishing arm64 support for 1.27 and newer - there will be no artifacts for older minors.

[philiplehmann]

could install it on my node with the air-gap install.

found two images which are not yet available

• rancher/system-agent-installer-rke2
• rancher/rke2-upgrade

[briandowns]

The scope of the arm work was purely RKE2. Further work will be done and is planned for rancher integration, etc.

[liyang516]

@liyang516 an experimental package of what? The release artifacts for arm64 are on github now, what else are you looking for?

Note that we are only publishing arm64 support for 1.27 and newer - there will be no artifacts for older minors.

tks

[liyang516]

could install it on my node with the air-gap install.

found two images which are not yet available

• rancher/system-agent-installer-rke2
• rancher/rke2-upgrade

What operating system did you use to install it?

[liyang516]

@philiplehmann What operating system did you use to install it?

[philiplehmann]

@philiplehmann What operating system did you use to install it?

running on an Ubuntu 22.04.2 LTS

used the guide from https://docs.rke2.io/install/airgap#rke2-installsh-script-install

mkdir /root/rke2-artifacts && cd /root/rke2-artifacts/
wget https://github.com/rancher/rke2/releases/download/v1.27.3%2Brke2r1/rke2-images.linux-arm64.tar.gz
wget https://github.com/rancher/rke2/releases/download/v1.27.3%2Brke2r1/rke2.linux-arm64.tar.gz
wget https://github.com/rancher/rke2/releases/download/v1.27.3%2Brke2r1/sha256sum-arm64.txt
curl -sfL https://get.rke2.io --output install.sh
INSTALL_RKE2_TYPE=agent INSTALL_RKE2_ARTIFACT_PATH=/root/rke2-artifacts sh install.sh 

[felixscheinost]

I think there is still an issue with Cilium ARM64 support in v1.27.3+rke2r1.

A new ARM64 node was stuck in NotReady state as rancher/hardened-cni-plugins:v1.0.1-build20221011 pull failed as it doesn't contain an image for arm64 architecture.

Workaround is to create a HelmChartConfig which updates the image:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    hubble:
      enabled: true
      relay:
        enabled: true
      ui:
        enabled: true
    portmapPlugin:
        image:
            repository: "rancher/hardened-cni-plugins"
            tag: "v1.2.0-build20230523"

[caroline-suse-rancher]

Hello everyone! The initial work on ARM64 support is done - as you encounter bugs or have enhancement requests, please open new issues, so we can appropriately track them. I'll be closing this issue as complete now.