Add nftables support #285

telmich · 2019-05-02T21:17:03Z

Currently jool does not work with nftables. However some networks switched away from ipXtables completely and thus jool cannot be used in those environments anymore.

I can be a beta tester, if there is a need for it.

ydahhrk · 2019-05-02T21:46:35Z

some networks switched away from ipXtables completely and thus jool cannot be used in those environments anymore.

Is Netfilter Jool useless as well?
https://jool.mx/en/intro-jool.html#design

ydahhrk · 2020-01-07T23:22:57Z

I keep needing to reference the original emails when talking about this bug, so here are some quick links to them:

(question) https://mail-lists.nic.mx/pipermail/jool-list/2019-May/000247.html
(response) https://mail-lists.nic.mx/pipermail/jool-list/2019-May/000248.html

telmich · 2020-01-08T09:03:21Z

If you need someone for testing -- just let me know!

It works but is also fairly flawed. Notable areas of improvement: - nft is not extensible, so adding the jool rule requires a custom nftables build. - Just like Netfilter Jool and iptables Jool, nftables Jool is jumping packets to postrouting, but since it hooks itself to the inet family, this might no longer be necessary. (Emphasis on "might.") - There are several unimplemented interface functions, such as nft_expr_ops.clone. Dunno what these do, but it probably wouldn't hurt to add them. - I'm currently testing on the filter chain, which is obviously not intended for this. Big chunk of #285.

ydahhrk · 2020-01-16T23:48:38Z

I have a functioning prototype, in case you want to hammer it.

Unfortunately, nft is not extensible, which means that I had to mess with its code instead of creating a plugin. For the moment, you need to compile a custom nft build to be able to use Jool's nftables wrapper. I will shortly interact with the Netfilter team and hopefully the situation will be improved.

Download, compile and install the custom nftables build (BTW: probably uninstall the old one beforehand):

(Edit 2020-04-28: Updated instructions to account for the new "jool" branches)

# I'm assuming you already have the autotools.
sudo apt install flex bison asciidoc libgmp-dev libreadline-dev libmnl-dev

git clone https://github.com/ydahhrk/libnftnl.git
cd libnftnl
git checkout jool
./autogen.sh
./configure
make
sudo make install
cd ..

git clone https://github.com/ydahhrk/nftables.git
cd nftables
git checkout jool
./autogen.sh
./configure
make
sudo make install
cd ..

sudo ldconfig

(I'm testing in Ubuntu 18.04.3, by the way.)

Download, compile and install the new Jool (BTW: probably uninstall the old one beforehand):

git clone https://github.com/NICMx/Jool.git
cd Jool
git checkout issue285
./autogen.sh
./configure
make
sudo make install
sudo dkms install .
# jool --version should print "4.0.7.2"

Basic SIIT recipe:

sudo modprobe jool_siit
sudo jool_siit instance add "nft-instance" --iptables -6 64:ff9b::/96

# the names of the table and chain are irrelevant, of course
sudo nft add table inet table1
sudo nft add chain inet table1 chain1 \{ type filter hook prerouting priority 0 \; \}
# first argument is the xlator type, second one is the instance name
sudo nft add rule  inet table1 chain1 jool siit "nft-instance"

Basic NAT64 recipe:

sudo modprobe jool
sudo jool instance add "nft-instance" --iptables -6 64:ff9b::/96

sudo nft add table inet table1
sudo nft add chain inet table1 chain1 \{ type filter hook prerouting priority 0 \; \}
sudo nft add rule  inet table1 chain1 jool nat64 "nft-instance"

That translates in my test environment.

Two notes:

About --iptables: From Jool's point of view, nftables instances are literally the same as iptables instances, hence the flag. I'm considering deprecating --iptables in favor of something like "--tables" instead. Suggestions welcomed.
I'm using the filter chain thingy, which is maybe wrong. I think the nat chain wouldn't work because it's defined to only accept the first packet of each stream, which is bad for Jool as currently implemented. But I haven't tested it.

How does it look?

ydahhrk · 2020-02-18T15:25:36Z

Here's the conversation in Netfilter Devel so far:

stv0g · 2020-04-05T06:56:10Z

Just for reference in case somebody else stumbles over this bug.

I am getting the following error message on CentOS 8 which is like caused because CentOS 8 switches to nftables:

Error: Error receiving the kernel module's response: Invalid input data or parameter

stv0g · 2020-04-05T23:02:26Z

@ydahhrk Your new nft support works like a charm 😁 👍

One idea to avoid patching nft: is it possible to incorporate the modification of the nft chains into the jool userspace binary?

ydahhrk · 2020-04-06T23:31:16Z

@ydahhrk Your new nft support works like a charm

@stv0g Thank you. With this bit of confirmation, and since it's pretty safe to say the mail thread was forgotten, I intend to upload a formal patch as another ping attempt tomorrow.

One idea to avoid patching nft: is it possible to incorporate the modification of the nft chains into the jool userspace binary?

Wouldn't it be problematic from a user's perspective? If nft doesn't understand Jool, this is the kind of stuff that happens:

$ # Prepare a table with a bunch of rules.
$ sudo nftables/src/nft add table inet graybox
$ sudo nftables/src/nft add chain inet graybox test \{ type filter hook prerouting priority 0 \; \}
$ sudo nftables/src/nft add rule inet graybox test ct state invalid drop
$ sudo nftables/src/nft add rule inet graybox test jool siit default
$ sudo nftables/src/nft add rule inet graybox test ct state invalid drop
$
$ # Print table, according to jool-aware nft
$ sudo nftables/src/nft list ruleset
table inet graybox {
	chain test {
		type filter hook prerouting priority filter; policy accept;
		ct state invalid drop
		jool "siit" "default"
		ct state invalid drop
	}
}
$
$ # Print table, according to jool-unaware nft
$ sudo nft list ruleset
table inet graybox {
	chain test {
		type filter hook prerouting priority filter; policy accept;
		ct state invalid drop

		ct state invalid drop
	}
}
netlink: Error: unknown expression type 'jool'.

stv0g · 2020-04-07T10:43:41Z

Oh yes. Thats a problem 👎

I had a brief look at the nftables-aware version of iptables. But I could't figure out if they support extensions.

ydahhrk · 2020-04-07T18:24:07Z

Poke sent:

I am getting the following error message on CentOS 8 which is like caused because CentOS 8 switches to nftables:

Error: Error receiving the kernel module's response: Invalid input data or parameter

What are you trying to do when this happens?

stv0g · 2020-04-07T23:17:39Z

What are you trying to do when this happens?

Adding an instance via jool instance

ydahhrk · 2020-04-08T14:38:39Z

Adding an instance via jool instance

Hmm... can you be more specific? I think this might be a bug. The error message is too generic, I can't find where it's being thrown, and I don't see how nft could be influencing it.

Can you still reproduce it?
Might I have the versions of your kernel modules and userspace clients? Example:

$ sudo modprobe -r jool
$ sudo modprobe jool
$ dmesg -t | tail -1 # kernel module version
NAT64 Jool v4.0.8.0 module inserted.
$ jool --version # userspace client version
4.0.8.0

Also the arguments to the instance add command.

stv0g · 2020-04-12T19:51:45Z

@ydahhrk I cant reproduce my previous problem anymore. I suspect it was due to a version mismatch as you mentioned..

stv0g · 2020-04-12T19:52:36Z

I found a segfault in the netfilter code somewhere.
It happens when I try to list the nft tables / chains:

(gdb) run list tables
Starting program: /usr/local/sbin/nft list tables
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-72.el8.x86_64
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
table ip nat
table ip filter
table ip mangle
table ip6 nat
table ip6 filter
table ip6 mangle
table ip6 raw
table ip raw
table inet jool-k8s-ziRE5S6H

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6eef133 in free () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install gmp-6.1.2-10.el8.x86_64 libmnl-1.0.4-6.el8.x86_64 ncurses-libs-6.1-7.20180224.el8.x86_64 readline-7.0-10.el8.x86_64
(gdb) bt
#0  0x00007ffff6eef133 in free () from /lib64/libc.so.6
#1  0x00007ffff7b60f6d in jool_stmt_destroy (stmt=0x74ccb0) at statement.c:677
#2  0x00007ffff7b62108 in stmt_free (stmt=0x74ccb0) at statement.c:51
#3  0x00007ffff7b621d8 in stmt_list_free (list=list@entry=0x619080) at statement.c:61
#4  0x00007ffff7b5cde0 in rule_free (rule=0x618eb0) at rule.c:631
#5  0x00007ffff7b5d79e in rule_free (rule=<optimized out>) at rule.c:864
#6  chain_free (chain=0x611010) at rule.c:864
#7  0x00007ffff7b5d876 in chain_free (chain=<optimized out>) at rule.c:1245
#8  table_free (table=0x607630) at rule.c:1245
#9  0x00007ffff7b5d9a1 in table_free (table=<optimized out>) at rule.c:1242
#10 __cache_flush (table_list=table_list@entry=0x605308) at rule.c:299
#11 0x00007ffff7b5d9cd in cache_release (cache=cache@entry=0x605300) at rule.c:305
#12 0x00007ffff7b86082 in nft_ctx_free (ctx=0x605260) at libnftables.c:294
#13 0x00000000004017f8 in main (argc=3, argv=0x7fffffffe598) at main.c:400

Jool statements are used to send packets to the Jool kernel module, which is an IP/ICMP translator: www.jool.mx Sample usage: modprobe jool jool instance add "name" --iptables -6 64:ff9b::/96 sudo nft add rule inet table1 chain1 jool nat64 "name" This feature was requested in Jool's bug tracker: NICMx/Jool#285 Signed-off-by: Alberto Leiva Popper <[email protected]>

Jool statements are used to send packets to the Jool kernel module, which is an IP/ICMP translator: www.jool.mx This feature was requested in Jool's bug tracker: NICMx/Jool#285 Signed-off-by: Alberto Leiva Popper <[email protected]>

ydahhrk · 2020-04-13T21:45:41Z

@stv0g Try the following branches:

https://github.com/ydahhrk/libnftnl/tree/jool
https://github.com/ydahhrk/nftables/tree/jool

stv0g · 2020-04-14T00:41:11Z

@ydahhrk Thanks the new branches fixed the segfault for me 👍

ydahhrk · 2020-04-28T23:39:40Z

Status update:

So the forecast is: My quick patch will not be accepted upstream.

In order to reconcile Jool and nftables, nothing less than a full merge will be required. This means this issue and #273 are one and the same.

Since this issue suddenly became a massive undertaking, my longterm plan is to finish #136 (because it's already in the testing phase), then implement #193, and then #285/#273.

(This excludes emerging urgent minor patches such as #326 and #325, which of course need to be addressed first.)

ydahhrk · 2020-04-28T23:45:34Z

Since this issue became a duplicate of #273, I will now close it.

In case anyone wants a quick 'n dirty version of this feature, my prototype is still available in my account and, as far as I know, is functional:

ydahhrk added the New feature label May 2, 2019

ydahhrk added the Status: Not Started Progress is 0% label Aug 28, 2019

ydahhrk mentioned this issue Dec 24, 2019

Merge Jool into nftables #273

Open

ydahhrk added Status: Active Development is underway and removed Status: Not Started Progress is 0% labels Jan 15, 2020

ydahhrk closed this as completed Apr 28, 2020

ydahhrk removed the Status: Active Development is underway label Apr 28, 2020

rnhmjoj mentioned this issue Jul 8, 2023

nixos/jool: add service for setting up SIIT/NAT64 NixOS/nixpkgs#240982

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nftables support #285

Add nftables support #285

telmich commented May 2, 2019

ydahhrk commented May 2, 2019

ydahhrk commented Jan 7, 2020 •

edited

Loading

telmich commented Jan 8, 2020

ydahhrk commented Jan 16, 2020 •

edited

Loading

ydahhrk commented Feb 18, 2020

stv0g commented Apr 5, 2020

stv0g commented Apr 5, 2020

ydahhrk commented Apr 6, 2020

stv0g commented Apr 7, 2020

ydahhrk commented Apr 7, 2020

stv0g commented Apr 7, 2020

ydahhrk commented Apr 8, 2020 •

edited

Loading

stv0g commented Apr 12, 2020

stv0g commented Apr 12, 2020

ydahhrk commented Apr 13, 2020

stv0g commented Apr 14, 2020

ydahhrk commented Apr 28, 2020 •

edited

Loading

ydahhrk commented Apr 28, 2020

Add nftables support #285

Add nftables support #285

Comments

telmich commented May 2, 2019

ydahhrk commented May 2, 2019

ydahhrk commented Jan 7, 2020 • edited Loading

telmich commented Jan 8, 2020

ydahhrk commented Jan 16, 2020 • edited Loading

ydahhrk commented Feb 18, 2020

stv0g commented Apr 5, 2020

stv0g commented Apr 5, 2020

ydahhrk commented Apr 6, 2020

stv0g commented Apr 7, 2020

ydahhrk commented Apr 7, 2020

stv0g commented Apr 7, 2020

ydahhrk commented Apr 8, 2020 • edited Loading

stv0g commented Apr 12, 2020

stv0g commented Apr 12, 2020

ydahhrk commented Apr 13, 2020

stv0g commented Apr 14, 2020

ydahhrk commented Apr 28, 2020 • edited Loading

ydahhrk commented Apr 28, 2020

ydahhrk commented Jan 7, 2020 •

edited

Loading

ydahhrk commented Jan 16, 2020 •

edited

Loading

ydahhrk commented Apr 8, 2020 •

edited

Loading

ydahhrk commented Apr 28, 2020 •

edited

Loading