[BUG] pfctl still invoked on VNET jails

[michael-o]

[MANDATORY] Describe the bug [MANDATORY]
I have a vnet jail, no loopback bastille0 yet I see on stop:

# bastille restart deblndw013x1j
pfctl: /dev/pf: No such file or directory
rdr-anchor not found in pf.conf
[deblndw013x1j]:
deblndw013x1j: removed

I don't have pf enabled and don't need it on the jailhost.

[MANDATORY] Bastille and FreeBSD version (paste bastille -v && freebsd-version -kru output)
0.8.20210115
12.2-STABLE
12.2-STABLE
12.2-STABLE

[MANDATORY] How did you install bastille? (port/pkg/git)
port

[optional] Steps to reproduce?
Create a vnet jail, start and stop

[optional] Expected behavior
No pfctl actions

[optional] Additional context
The issue comes from here:

bastille/usr/local/share/bastille/stop.sh

Lines 59 to 60 in af8d766

	if [ "$(bastille rdr ${_jail} list)" ]; then
	bastille rdr ${_jail} clear

My understanding ist that pf is only necessary when you don't have external external IP addresses and need to do NAT. This does not apply to VNET jails. I do run VNET only. I don't know whether bastille rdr ... can also be run at runtime, but all rdr/pdf/loopback related checks should be skipped for a VNET jail.

[KizzyCode]

That depends; for example I have a VNET setup which is NATed to my external interface so that I can access the internet or deploy a VPN server inside a jail.

So in my case it's a good thing that Bastille behaves this way, however there should be a way to configure this depending on your setup I guess.

[michael-o]

That depends; for example I have a VNET setup which is NATed to my external interface so that I can access the internet or deploy a VPN server inside a jail.

What is the purpose of having VNET jails when you do NAT where you could just use cloned lo1? Isn't the whole point being not dependent of the host's interface?

So in my case it's a good thing that Bastille behaves this way, however there should be a way to configure this depending on your setup I guess.

Maybe for you, but this is an assumption Bastille does w/o knowning this jail uses pf anyhow. It should be configure whether bastille should interact with pf.

[KizzyCode]

What is the purpose of having VNET jails when you do NAT where you could just use cloned lo1? Isn't the whole point being not dependent of the host's interface?

Some applications require a dedicated network stack and do not work with lo; e.g. WireGuard or some firewall applications. Therefore you need to create a VNET that is bridged to your external interface for inet connectivity. And because you usually have only one IPv4 address, you need NAT and RDR.

[michael-o]

@KizzyCode I'd be interested how this looks like configuration-wise.

[KizzyCode]

Ok, basically:

1. You create a bridge device; e.g. em0bridge and add your external interface em0 to it
2. You assign an IP to your bridge that serves as gateway; e.g. 10.0.1.1/24
3. You create an epair pair; e.g. vnet0_host and vnet0_jail (bastille does that automatically)
4. You add vnet0_host to em0bridge (bastille does that automatically)
5. You add vnet0_jail to the jail and assign a subnet compatible address; e.g. 10.0.1.10/24 (bastille does that automatically)
6. You add your bridge's address 10.0.1.1 as default route to the jail

So thats the point where we need NAT for outgoing traffic and RDR for incoming traffic. Currently I use nat on em0 from em0bridge:network to any -> em0; the redirects are configured via pf (using rdr-anchor "rdr/*" in the config).

For a more complete example, take a look at: https://genneko.github.io/playing-with-bsd/networking/freebsd-wireguard-jail/

[michael-o]

While I see -- after @KizzyCode's explanation -- use cases for this, this should be create time option whether bastiile should take care of this or not.

[michael-o]

Can this finally be solved by provinding a --pf or something similar to the bastille create command? Bastille can memorize it and call pf only if the jail has been created with the option.

[JRGTH]

I don't use pf and I have a similar jail stop issue here, however I'm not sure but I think this PR: 211a268 solves the stop pf invocation issue, as well as for embedded system only using IPFW or not having pf command at all.

Regards

[michael-o]

Can this finally be addressed?

[michael-o]

This is still open in the latest relese :-(

[tschettervictor]

Is this still an issue?
I'm trying to reproduce it, but can't.
@michael-o

The start and stop commands will only invoke pfctl if the command itself is found, or the "ip4.addr" is set in jail.conf

[michael-o]

Is this still an issue? I'm trying to reproduce it, but can't. @michael-o

The start and stop commands will only invoke pfctl if the command itself is found, or the "ip4.addr" is set in jail.conf

The command is always present:

root@deblndw013x:~
# which pfctl
/sbin/pfctl
root@deblndw013x:~
# service pf status
pf.ko is not loaded

thus

root@deblndw013x:~
# bastille restart deblndw013x2j
pfctl: /dev/pf: No such file or directory
rdr-anchor not found in pf.conf
[deblndw013x2j]:
deblndw013x2j: removed

pfctl: /dev/pf: No such file or directory
[deblndw013x2j]:
e0a_bastille3
e0b_bastille3
deblndw013x2j: created

with

root@deblndw013x:~
# bastille version
0.12.20241124

[tschettervictor]

Can you try adding "> /dev/null" to the "which -s pfctl" command?

[michael-o]

root@deblndw013x:~
# which -s pfctl ; echo $?
0

[tschettervictor]

I mean inside the "stop.sh" script.

[michael-o]

I mean inside the "stop.sh" script.

Seems to be inconsistent this check:

root@deblndw013x:/usr/local/share/bastille
# grep  pfctl start.sh stop.sh
start.sh:            pfctl -q -t "${bastille_network_pf_table}" -T add "${ip}"
stop.sh:        # Check if pfctl is present
stop.sh:        if which -s pfctl; then
stop.sh:                pfctl -q -t "${bastille_network_pf_table}" -T delete "${_ip}"

but does one expect the system to be compiled without pf? The test with which does not indicate that pdf is actually running...

[tschettervictor]

This check should actually check if NAT is being used before attempting to call pfctl.

[michael-o]

This check should actually check if NAT is being used before attempting to call pfctl.

Obviously, it doesn't :-(

[michael-o]

Here is the problem:

root@deblndw013x:/usr/local/share/bastille
# bastille config deblndw013x1j get ip4.addr
not set

but the test does:

 82         ip=$(bastille config "${_jail}" get ip4.addr)
 83         if [ -n "${ip}" ]; then

obviously bastille config is implemented incorrectly

[michael-o]

The logic to determine the IPv4 addressed in start.sh is different to stop.sh. stop.sh needs to be aligned to start.sh and the issue should be gone.

[michael-o]

Fix for start.sh:

if [ "${ip}" != 'not set' ]; then

Will need to raise a PR for this.

[tschettervictor]

Fix for start.sh:

if [ "${ip}" != 'not set' ]; then

Will need to raise a PR for this.

Does this function as expected?

[michael-o]

Fix for start.sh:

if [ "${ip}" != 'not set' ]; then

Will need to raise a PR for this.

Does this function as expected?

Yes, pf is then skipped at start time.

[tschettervictor]

Great! I'll open a PR.

[tschettervictor]

For me the ip4.addr can also sometimes be "-" so I'll include that also.

[michael-o]

For me the ip4.addr can also sometimes be "-" so I'll include that also.

How so? Buy if so, search for other occurrences as well.

[tschettervictor]

root@webmin:~ # jls -j jail2 ip4.addr
-
root@webmin:~ # jls -j jail2 ip6.ddr
-

My bad. That's just when using the raw command.
Bastille returns "not set"

[tschettervictor]

#769

[bmac2]

fixed with PR just merged

[dbolser]

If I want to use vnet, does it do any harm to have pf running with

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

in the pf.conf?