Odroid C2

[DorianGray]

So, I've made a stripped down image of ubuntu(2.2gb) that works on odroid c2 and hacked the scripts here together to make it work. I was surprised at how little compilation I had to do, ubuntu's aarch64 support is stellar. I have a couple more things to do, mostly cleanup, but almost there! I'll host the image somewhere like s3, or maybe github if they'll let me once it's done(the official image's kernel is insufficient and is more than twice as big as it should be).

I'm thinking about changing out the root filesystem for btrfs and having docker just use subvolumes. Beyond that, maybe enabling compression to increase disk bandwidth. This thing has ample cpu power but not so much disk space or bandwidth. I dug into it and it looks like it will be a good chunk of work + testing to refactor things to allow for a configurable storage driver. Before I start, would that be something that you would want in the core scripts or should I just keep the main version using overlayfs?

As an aside, it still uses armhf kubernetes binaries but I built an aarch64 docker 1.10.2 using go1.6 on the machine itself. I have my cluster's master on armhf and a single aarch64 worker. I tagged the node as arch=aarch64, added a nodeSelector: arch: aarch64 for a 64bit jenkins and presto, it all worked! I guess I should have assumed cross architecture wouldn't be a problem, but I was very happy when it just worked.

[luxas]

Which kernel version do you have? The newer, the better. Arch Linux ARM has 3.14
Use overlayfs if possible, if not, check how I did the parallella.sh file in dynamic-env that swaps the overlay driver for something that works.

We should use armhf binaries for the time being for arm64 too, but after some releases we might use k8s arm64 ones.

[DorianGray]

Kernel is 3.14 with several docker specific features backported. I just had to enable a few flags extra to work with aufs and overlay...overlay is still giving me trouble, but I'll see if I can't get it working.

[DorianGray]

Btw, theoretically the rasp pi 3 could work on this image with some /boot modifications... they do use the same cpu.

[DorianGray]

I am going to have to use btrfs for performance and reliability reasons. AUFS is too slow and wonky and overlay doesn't work in this kernel...

Even after having it patched and the module loaded docker won't recognize the overlay driver. It probably has a hard check on kernel version or is looking for a slightly different name or something weird.

[DorianGray]

#75

[larmog]

I got, just now, Docker docker-hypriot_1.10.3-1_armhf.deb working on C2 Odrobian (Debian based) with 32-bit (armhf) support. Maybe that's an better alternative?

[DorianGray]

No aarch64 support for applications though =(

[luxas]

Why not just Arch Linux?
And I saw that you @larmog has tested out Hypriot too. Success or failure?

[DorianGray]

Arch linux 64 sounds like it will work. It would need to use the odroid c2 kernel and I'm not sure what, if any, uboot modifications would be required. Personally, I don't want to invest the time in it right now. It works on ubuntu now for my purposes, and I am willing to spend the time to support rancheros aarch64 in the future because I think that's the long term solution for this project.

[larmog]

@luxas I couldn't get the Hypriot image to boot. It hangs in U-boot.
I wanted an image that supported both armhf and aarch64 (multiarch) and found Odrobian. I've verified that it runs both armhf and aarch64 images and it works with 0.7.0-dev.

Arch Linux should work, we just need to enable multiarch, until there's a 64-bit build of the binaries.

[luxas]

I'm working on upstream binaries for arm64. It depends though on kubernetes/kubernetes#22149 and some other things.

If you want to try out arm64 binaries anyway, I quickly hacked a script for that: https://github.com/luxas/kubernetes-on-arm/blob/dev/scripts/build-k8s-on-amd64/Dockerfile.arm64. Haven't tested it yet.

I'll have to think about how to use/implement #75 and #79 in the best way. I want Odroid support to this project although I haven't one (yet). But I want it independently maintained like Arch Linux or HypriotOS

[larmog]

@luxas I reviewed the code for Arch. I didn't know you used your own static compiled version of Docker. I use hypriotos and used your kubernetes-on-arm deb package.

I can't see any problems building an image for Arch and C2 the same way. Personally I would prefer to use the package manager for handling binaries like Docker.

I think it would be beneficial (initially) to have an image that support both armhf and aarch64 in cases (like me) where you have a mixed architecture.

I'll try to add Arch support for C2, if I can find the time. I'm sure the Hypriot team will release a C2 image of hypriotos in a near future.