Port k3s to Android 14

[bobmorane83]

K3s Versions to be Validated

# k3s --version
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2

Hi all,

I'am trying to run k3s over a real Android 14 arm64 hardware.
All kernel flags have been enabled and generic Linux directories added. BTW, I already run Docker successfully (root, no SELinux. no overlayfs)

Running k3s

k3s server -o /data/rancher/config --config /data/rancher/config.yaml

I have this "INFO":

INFO[0015] Waiting to retrieve agent configuration; server is not ready: "overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native": failed to mount overlay: invalid argument 

and I get :

# kubectl get node 
NAME        STATUS     ROLES                       AGE   VERSION
localhost   NotReady   control-plane,etcd,master   28h   v1.30.1+k3s1

So it seems that the "mount" command to mount overlayfs path do not have correct parameters on Android.
Maybe this might be due to F2FS used by Android as main FS. I don't have enough knowledge for now on this FS and k3s.

Adding --snapshotter=native gives:

E1009 13:04:09.807103   17872 kubelet.go:1547] "Failed to start ContainerManager" err="failed to get rootfs info: failed to get mount point for device \"/dev/block/dm-6\": no partition info for device \"/dev/block/dm-6\""

For the record, to run k3s successfully on Cuttlefish x86, I had to change the FS to EXT4 and use --snapshotter=native.

BTW,
I can't cross re-compile k3s for arm64 on x86 PC (to add more debug logs).
My cmd are :

mkdir -p build/data && ARCH=arm64 make download && ARCH=arm64 make generate
ARCH=arm64 SKIP_VALIDATE=true make

the k3s is still for x86

Any help will be appreciated,
Tx,

[bobmorane83]

Adding fuse-overlayfs with option --snapshotter=fuse-overlayfs I still get an error :

INFO[0010] Waiting to retrieve agent configuration; server is not ready: "fuse-overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "native": fuse-overlayfs not functional, make sure running with kernel >= 4.18: failed to mount fuse-overlayfs ({Type:fuse3.fuse-overlayfs Source:overlay Target: Options:[lowerdir=/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower2:/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower1]}) on /var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/merged: mount helper [mount.fuse3 [overlay /var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/merged -o lowerdir=/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower2:/var/lib/rancher/k3s/agent/containerd/fuseoverlayfs-check1358862875/lower1 -t fuse-overlayfs]] failed: "unknown argument ignored: lazytime\nunknown argument ignored: -t\nfuse: failed to access mountpoint fuse-overlayfs: No such file or directory\nfuse-overlayfs: cannot mount: No such file or directory\n": exit status 1

Even if I can create a working overlay with something like :
fuse-overlayfs -o lowerdir=$PWD/lower,workdir=$PWD/work,upperdir=$PWD/upper $PWD/overlay

Seems arguments are not correct ...

[bobmorane83]

Using Docker (version 20.10.21, containerd v1.6.9) with option --docker lead to the same error from F2FS :

E1009 13:04:09.807103   17872 kubelet.go:1547] "Failed to start ContainerManager" err="failed to get rootfs info: failed to get mount point for device \"/dev/block/dm-6\": no partition info for device \"/dev/block/dm-6\""

[brandond]

i suspect f2fs is lacking support some of the sysctls required for this to work properly. You'd probably have to make an xfs or ext3 partition available.

[bobmorane83]

i suspect f2fs is lacking support some of the sysctls required for this to work properly. You'd probably have to make an xfs or ext3 partition available.

Yes, using an ext4 partition is Ok.
I'll investigate what is missing.

[liyimeng]

@bobmorane83 I guess that you are at least need to mount /var/lib/rancher/k3s/agent on top of a ext4/xfs partition.

Do you use Android 14 vanilla or a customize one?

[tomaszduda23]

I run k3s agent #11130

[bobmorane83]

I run k3s agent #11130

Thanks ! Very interesting. I'll investigate in this direction.
Did you tried to run k3s as server ?

From my side, I reached to run it on ext4 partition, but have the known "route default" problem as it seems that Android do not have default IP route checked by k3s (or dependancies).

BTW, do you cross compile k3s or do you compile it directly on arm64 device ?

BR.

[tomaszduda23]

I've not checked server. You don't need ext4 I run it directly on f2fs. I updated script in the other thread. It fixes "route default". I build directly on android.

[tomaszduda23]

tomaszduda23@259433c I recently update to this if anyone is interested.

[tomaszduda23]

btw for solving "default root" ndc could be used. It can control network configuration on Android. https://android.googlesource.com/platform/system/netd/+/master/server/NdcDispatcher.cpp. E.g. for docker it fix network partially

ndc network create 30
ndc network interface add 30 docker0
ndc network route add 30 docker0 172.17.0.0/16
ndc ipfwd add lo docker0

[tomaszduda23]

@brandond do you think that patches from tomaszduda23@259433c could be merged to k3s?

[cwayne18]

No.

[tomaszduda23]

would you mind to share some details why it would not be possible to merge? of course some additional work would be required.

I'm running k3s on android for a few months and it works great. updating kernel for lineageos supported device is quite easy. It requires a few changes to make it work out of the box:

• Android use f2fs which is not supported by tomaszduda23/cadvisor@acb9743
• cgroups are mounted with no prefix support cgroup v1 mounted with noprefix opencontainers/runc#4513
• containerd reads dns only from /etc/resolv.conf which do not exist on android
  [v1.31.1+k3s-452dbbc1] containerd reads ca from /etc/ssl/certs/ca-certificates.crt which do not exist on android
• cgroupv1 is read from random directory /dev or /sys. There is wired logic which decided which directory should be used.

[brandond]

Well first of all the replacements in go.mod aren't even properly formatted.

Second, I have no idea you are or why we should use your fork of runc and cadvisor instead of the upstream projects or a forks maintained by this project. I would encourage you to get your changes pushed upstream, at which point we could take advantage of them. We do not pull in random forks.

Third: https://en.wikipedia.org/wiki/Supply_chain_attack

[tomaszduda23]

PRs for upstream are already open opencontainers/runc#4513, google/cadvisor#3604. runc probably will be merged at some point. cadvisor I'm not sure.

Both are forked in k3s organization. I would like to open the same PRs to https://github.com/k3s-io/cadvisor and https://github.com/k3s-io/runc. If you would be willing to accept those PRs. In house forks could be used for build of k3s.

[brandond]

We are not accepting community patches into our forks at this time. We try very hard to only add the bare minimum necessary to make the components work when embedded in k3s; we are not doing extra to cover platforms that we can't provide testing or QA for. When the patches are accepted upstream, we can update our forks to track those changes.

See also our current policy on building for new architectures like riscv64. While we do want to encourage community contributions, we do also have to consider what we are able to actually maintain and support without someone on the core team to own it.