From b028190c7cfe38709381d064d26cea4ab0c05fee Mon Sep 17 00:00:00 2001 From: Alex Tomic Date: Wed, 7 Aug 2024 11:09:35 -0400 Subject: [PATCH] opir1 (armv7) support working with new BTF kernel build begin streamlining install scripts and docs --- README.md | 18 ++++--- doc/install.md | 18 +++---- ebpf/Makefile | 2 +- scripts/customize-image.sh | 105 +++++++++++++++++++++++++++++-------- 4 files changed, 102 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 14f68fb..96d8750 100644 --- a/README.md +++ b/README.md @@ -12,14 +12,16 @@ etc.) while avoiding the downsides of excessive use. ## Getting Started -Nethadone is in active development and only been tested on an -[Orange Pi R1 Plus](http://www.orangepi.org/orangepiwiki/index.php/Orange_Pi_R1_Plus). -If you are interested in giving it a try on a similar device, -please check out `doc/install.md` for details, feedback and -contributions are more than welcome! +Nethadone is in active development and has been tested on the following devices: + +* [Orange Pi R1 Plus](http://www.orangepi.org/orangepiwiki/index.php/Orange_Pi_R1_Plus) +* [Orange Pi R1](http://www.orangepi.org/orangepiwiki/index.php/Orange_Pi_R1) + +Any similar device using an armv7 or arm64 chipset should work. +Please check out [doc/install.md](doc/install.md) for details. For development or testing on a local VM, please see -`doc/development.md` +[doc/development.md](doc/development.md) ## How does it work? @@ -38,9 +40,9 @@ As a user continues to compulsively use a given site, traffic to flagged IPs gradually works its way down the list above. Nethadone currently only supports a single policy, for more details -see `policy/README.md`. +see [policy/README.md](policy/README.md). -For more details on bandwidth classes (i.e. qdisc configuration), see `doc/qdisc.md` +For more details on bandwidth classes (i.e. qdisc configuration), see [doc/qdisc.md](doc/qdisc.md). ### Traffic flow diff --git a/doc/install.md b/doc/install.md index dc23104..fe6f2da 100644 --- a/doc/install.md +++ b/doc/install.md @@ -18,12 +18,10 @@ Nethadone has been tested on: * 1GB RAM * 2x GBit ethernet * Wifi (not functional w/ Armbian yet) - -In progress: - * Orange Pi R1 * 32-bit Allwinner H3 * 2 x 100Mbit ethernet + * Wifi (working) * 256MB / 512MB RAM In theory, any device with two network interfaces should work, but I have not had a chance to test others. @@ -32,13 +30,18 @@ An SD card 16GB or greater is recommended. ### OS -Ubuntu 22.04 LTS or derivative (eg. Armbian 23.8) +Ubuntu-based 22.04 LTS or derivative (eg. Armbian 23.8) +Ubuntu 22.04 or 24.04 LTS and derivatives (i.e. Armbian) are working Other OS versions may work, but the further away from Linux 6.1 (either newer or older), the more likely eBPF issues may be encountered. -## OS Setup +## OS Setup - Orange Pi R1+ (Arm64) + +> [!NOTE] +> I am in the process of streamlining this installation process, +> first for the armv7 Orange Pi R1 and eventually for the R1+. ### Base image For the Orange Pi R1+, you need to get the Armbian 23.8.1 Jammy build: @@ -49,11 +52,6 @@ This is because we will need to use a BTF-enabled kernel provided by [daeuniverse](https://github.com/daeuniverse/armbian-btf-kernel), and that is the latest version of Armbian provided there. -A TODO item is to provide updated linux kernels for recent versions -of Armbian, or an easy to use build script using the Armbian -customize-image hook that preps the image plus the required kernel -options. - ### Image prep The contents of `scripts/customize-image.sh` will need to be run, diff --git a/ebpf/Makefile b/ebpf/Makefile index 19f8290..520596c 100644 --- a/ebpf/Makefile +++ b/ebpf/Makefile @@ -25,7 +25,7 @@ vmlinux.h: bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h build-throttler: - clang -g -O2 -I/usr/include/aarch64-linux-gnu -Wall -target bpf -c throttle.bpf.c -o throttle.o + clang -g -O2 -I/usr/include/aarch64-linux-gnu -I/usr/arm-linux-gnueabi/include -Wall -target bpf -c throttle.bpf.c -o throttle.o rtarg = dnspkt direction = ingress diff --git a/scripts/customize-image.sh b/scripts/customize-image.sh index a667729..0ff80da 100644 --- a/scripts/customize-image.sh +++ b/scripts/customize-image.sh @@ -1,25 +1,46 @@ #!/bin/bash -apt-get update +case "$(arch)" in + 'aarch64') + prom_url='https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-arm64.tar.gz' + go_url='https://go.dev/dl/go1.22.3.linux-arm64.tar.gz' + kern_url='https://github.com/daeuniverse/armbian-btf-kernel/releases/download/main-2023-06-17/kernel-rockchip64-current_23.08.0-trunk--6.1.34-Sca87-Dbeb1-Pa401-C3053Hfe66-HK01ba-Vc222-B76dc.tar' + ;; + 'armv7l') + prom_url='https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-armv7.tar.gz' + go_url='https://go.dev/dl/go1.22.3.linux-armv6l.tar.gz' + kern_url='https://github.com/atomic77/nethadone/releases/download/btf-kernel/kernel-legacy-sunxi-orangepi-r1.tar.gz' + ;; + 'x86_64') + prom_url='https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-amd64.tar.gz' + go_url='https://go.dev/dl/go1.22.3.linux-arm64.tar.gz' + # Most x86_64 builds for virtual machine testing use should have BTF enabled + '*') + echo "Unsupported architecture $(arch), exiting." + exit 1 + ;; -# For Debian bookworm -apt-get install -y apt-transport-https ca-certificates curl clang llvm jq \ - libelf-dev libpcap-dev libbfd-dev binutils-dev build-essential make \ - bpfcc-tools python3-pip vim libbpf-dev \ - avahi-daemon bcc python-is-python3 \ - python3-dnslib python3-cachetools # for tcpconnect dns tracing in python +esac -# For ubuntu jammy -apt-get install linux-tools-common -y +apt-get update -y -wget https://go.dev/dl/go1.22.3.linux-arm64.tar.gz -rm -rf /usr/local/go && tar -C /usr/local -xzf go1.22.3.linux-arm64.tar.gz +apt-get install -y apt-transport-https ca-certificates curl clang llvm jq \ + libelf-dev libpcap-dev libbfd-dev binutils-dev build-essential make \ + vim libbpf-dev avahi-daemon linux-tools-common dnsmasq +wget ${go_url} +rm -rf /usr/local/go && tar -C /usr/local -xzf go1.* +rm go*.tar.gz echo "export PATH=\$PATH:/usr/local/go/bin" >> /etc/profile +##### # Local prometheus for metrics collection -wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-arm64.tar.gz -tar -C /usr/local -xzf prometheus-2.52.0.linux-arm64.tar.gz +wget ${prom_url} +tar -C /usr/local/bin --strip-components 1 -xzf prometheus-*.tar.gz +rm prometheus-*.tar.gz + +cp /usr/local/bin/prometheus.yml /etc +mkdir -p /var/lib/prometheus cat >> /etc/prometheus.yml << EOF - job_name: "nethadone" @@ -27,17 +48,55 @@ cat >> /etc/prometheus.yml << EOF - targets: ["localhost:3000"] EOF -# TODO Add prometheus systemd unit, data directory for autostart -# TODO and ip masq setup / ipv4 forward on boot, or enable -# in nethadone directly, i.e. -# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -# sysctl net.ipv4.ip_forward=1 +cat < /etc/systemd/system/prometheus.service +[Unit] +Description=Prometheus +Wants=network-online.target +After=network-online.target + +[Service] +User=root +Restart=on-failure + +ExecStart=/usr/local/bin/prometheus \ + --config.file=/etc/prometheus.yml \ + --storage.tsdb.path=/var/lib/prometheus + +[Install] +WantedBy=multi-user.target + +EOF + +systemctl daemon-reload +systemctl enable prometheus + +###### +# NAT forwarding will need to be in place for routing to work, +# but the interface may change depending on your configuration. +# TODO Configure this in nethadone directly +#echo "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE" >> /etc/rc.local + +echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/20-nethadone.conf + + +# For wifi-enabled devices, network-manager does a nice job of setting up an AP +# that automatically proxies DNS requests, eliminating the need for pihole. +# eg: +# sudo nmcli dev wifi hotspot ifname wlan0 ssid nethadone password "mypass" +# ##### # Grab the custom BTF-enabled kernel from daeuniverse' repo -wget https://github.com/daeuniverse/armbian-btf-kernel/releases/download/main-2023-06-17/kernel-rockchip64-current_23.08.0-trunk--6.1.34-Sca87-Dbeb1-Pa401-C3053Hfe66-HK01ba-Vc222-B76dc.tar -tar xf kernel-rockchip64-current_23.08.0-trunk--6.1.34-Sca87-Dbeb1-Pa401-C3053Hfe66-HK01ba-Vc222-B76dc.tar -yum install ./linux-* +if [ $(arch) != 'x86_64' ]; then + wget ${kern_url} + tar xvf kernel-* + apt-get install -y ./linux-* + rm kernel*.tar +fi + +if [ $(arch) == 'armv7' ]; then + apt-get install libc6-dev-armel-cross -y +fi # Build BPFTool from source - repo packages don't seem to work properly mkdir ~/src @@ -46,6 +105,8 @@ git clone --recurse-submodules https://github.com/libbpf/bpftool.git cd bpftool/src make -j 4 make install +rm -rf ~/src - +# TODO - Copy in pre-built nethadone binary from github to avoid git +# checkout and golang compiler