ip6tables unknown option --set-xmark

[tristanlins]

After a recent OS Update of my Fedora 40 Servers, K3S does not start anymore.
I tried to start k3s with --prefer-bundled-bin, without success.

Environmental Info:
K3s Version:
k3s version v1.30.5+k3s1 (9b58670)
go version go1.22.6

Node(s) CPU architecture, OS, and Version:
Linux 6.11.4-201.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Oct 20 15:04:22 UTC 2024 x86_64 GNU/Linux
Fedora Linux 40 (Server Edition)
nftables-1.0.9-3.fc40
iptables-1.8.10-7.fc40

Cluster Configuration:
Single Node.

Describe the bug:
K3S failed to start with ip6tables v1.8.9 (legacy): unknown option "--set-xmark".

From the logs:

Okt 26 01:54:40 ... k3s[20378]: E1026 01:54:40.885702   20378 proxier.go:1511] "Failed to execute iptables-restore" err=<
Okt 26 01:54:40 ... k3s[20378]:         exit status 2: Ignoring deprecated --wait-interval option.
Okt 26 01:54:40 ... k3s[20378]:         Warning: Extension MARK revision 0 not supported, missing kernel module?
Okt 26 01:54:40 ... k3s[20378]:         ip6tables-restore v1.8.9 (legacy): unknown option "--xor-mark"
Okt 26 01:54:40 ... k3s[20378]:         Error occurred at line: 31
Okt 26 01:54:40 ... k3s[20378]:         Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
Okt 26 01:54:40 ... k3s[20378]:  >
...
Okt 26 01:54:43 ... k3s[20378]: F1026 01:54:43.919772   20378 network_policy_controller.go:637] Failed to run iptables command: running [/var/lib/rancher/k3s/data/ac0baecab6b7fa399482b08daa7117e7f2a0b1a739da5c31131bea4ebfaedfec/bin/aux/ip6tables -t filter -C KUBE-NWPLCY-DEFAULT -j MARK -m comment --comment rule to mark traffic matching a network policy --set-xmark 0x10000/0x10000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
Okt 26 01:54:43 ... k3s[20378]: ip6tables v1.8.9 (legacy): unknown option "--set-xmark"
Okt 26 01:54:43 ... k3s[20378]: Try `ip6tables -h' or 'ip6tables --help' for more information.
Okt 26 01:54:43 ... k3s[20378]: panic: F1026 01:54:43.919772   20378 network_policy_controller.go:637] Failed to run iptables command: running [/var/lib/rancher/k3s/data/ac0baecab6b7fa399482b08daa7117e7f2a0b1a739da5c31131bea4ebfaedfec/bin/aux/ip6tables -t filter -C KUBE-NWPLCY-DEFAULT -j MARK -m comment --comment rule to mark traffic matching a network policy --set-xmark 0x10000/0x10000 --wait]: exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module?
Okt 26 01:54:43 ... k3s[20378]: ip6tables v1.8.9 (legacy): unknown option "--set-xmark"
Okt 26 01:54:43 ... k3s[20378]: Try `ip6tables -h' or 'ip6tables --help' for more information.
Okt 26 01:54:43 ... k3s[20378]: goroutine 44617 [running]:
Okt 26 01:54:43 ... k3s[20378]: k8s.io/klog/v2.(*loggingT).output(0xaa950c0, 0x3, 0xc0010448e0, 0xc003347c00, 0x1, {0x86866ff?, 0x1?}, 0x0?, 0x0)
Okt 26 01:54:43 ... k3s[20378]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:965 +0x73d
Okt 26 01:54:43 ... k3s[20378]: k8s.io/klog/v2.(*loggingT).printfDepth(0xaa950c0, 0x3, 0xc0010448e0, {0x0, 0x0}, 0x1, {0x63633d9, 0x22}, {0xc013698fc0, 0x1, ...})
Okt 26 01:54:43 ... k3s[20378]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:767 +0x1f0
Okt 26 01:54:43 ... k3s[20378]: k8s.io/klog/v2.(*loggingT).printf(...)
Okt 26 01:54:43 ... k3s[20378]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:744
Okt 26 01:54:43 ... k3s[20378]: k8s.io/klog/v2.Fatalf(...)
Okt 26 01:54:43 ... k3s[20378]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:1655
Okt 26 01:54:43 ... k3s[20378]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureDefaultNetworkPolicyChain(0xc009a3efc0?)
Okt 26 01:54:43 ... k3s[20378]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:637 +0x4ce
Okt 26 01:54:43 ... k3s[20378]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).Run(0xc009a3efc0, 0xc00e4ac5a0, 0xc001afca20, 0xc00e3942e0)
Okt 26 01:54:43 ... k3s[20378]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:172 +0x17e
Okt 26 01:54:43 ... k3s[20378]: created by github.com/k3s-io/k3s/pkg/agent/netpol.Run in goroutine 1
Okt 26 01:54:43 ... k3s[20378]:         /go/src/github.com/k3s-io/k3s/pkg/agent/netpol/netpol.go:184 +0xe34

Steps To Reproduce:

• Installed K3s: curl -sfL https://get.k3s.io | sh -s - [--prefer-bundled-bin]

Additional context / logs:

• k3s.log

[EAJ-EAJGlobal]

It's a kernel Issue, I was having the same issue for the past few days, came across this tailscale/tailscale#13863 where tailscale vpn are having the same issue, booting with an older kernel than 6.11.4 fixed it for me. Backported kernels may also suffer so may need to test other kernels to get it working, remove the one that's not working and prevent it updating until K3S updated the way they rely on ip6tables or a newer kernel is pushed. (Looks like it's a reported bug in the kernel repo)

[brandond]

Direct link to fix for the kernel bug: https://lore.kernel.org/all/[email protected]/

I doubt any projects are going to stop using xmark due to this; you'll need to upgrade or downgrade to an unaffected kernel version.

Closing this since it's not something we can fix in k3s, but feel free to keep discussing here.

[tristanlins]

Thanks, it was a pretty obvious solution to downgrade the kernel, huh? 🙈
I couldn't find much else that was helpful. Hopefully this issue can help others who run into the same problem. 😇

[tristanlins]

I have one more question. Is it possible that this issue is specific to IPv6? I've been wondering why my home server, which runs a kernel 6.11.4-101.fc39.x86_64 without IPv6, doesn't experience this problem. k3s starts just fine on that server.

[Kaurin]

Suffering from the same issue on Fedora

I have one more question. Is it possible that this issue is specific to IPv6? I've been wondering why my home server, which runs a kernel 6.11.4-101.fc39.x86_64 without IPv6, doesn't experience this problem. k3s starts just fine on that server.

I can't find in the documentation how to run a cluster without ipv6. I only found this.

I have disabled ipv6 on my system:

$ cat /etc/sysctl.d/70-disable-ipv6.conf 
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
net.ipv6.conf.all.disable_ipv6=1

$ sudo ip a | grep inet6
<no output>

I would very much appreciate it if you could share how to disable ipv6 in k3s

[brandond]

I would probably just downgrade the kernel.

[tristanlins]

I would very much appreciate it if you could share how to disable ipv6 in k3s

As far as I know, you can only choose between IPv4, IPv4+IPv6 (DualStack), and IPv6-only when you initially set up the cluster. My home server was deployed before k3s offered IPv6 support. Downgrading the kernel would probably be the simplest solution to this issue.

https://docs.k3s.io/networking/basic-network-options#dual-stack-ipv4--ipv6-networking
https://fedoramagazine.org/boot-earlier-kernel/

[slaecker]

I ran into this issue yesterday when creating a dualstack k3s cluster on openSUSE MicroOS running kernel 6.6.57-1-longterm.

I was also able to reproduce it on kernel 6.11.5-1-default but since today's transactional update k3s starts normally. The kernel version hasn't changed but comparing the snapshots I see that the kernel package was changed, so I guess it was patched.

The issue still occurs on 6.6.58-1-longterm though which was applied today as well.

EDIT: Downgrading to 6.6.56-1-longterm makes the issue disappear.

[venc0r]

Fedora CoreOS 41.20241027.3.0 (6.11.5-300.fc41.x86_64) is effected too, rollback to Fedora CoreOS 40.20241019.3.0 (6.11.3-200.fc40.x86_64) helps

[venc0r]

Fedora CoreOS 41.20241109.3.0 (6.11.6-300.fc41.x86_64) has it fixed