Bastille console does not work after upgrading jail to 14.1-RELEASE

[paulz-gh]

[MANDATORY] Describe the bug [MANDATORY]
I upgraded my system from 13.3 to 14.1-RELEASE, including pkg-static update -f.
I have run bastille bootstrap 14.1-RELEASE.

When I update a jail by changing the mount from 13.3-RELEASE to 14.1-RELEASE the jail seems to start but the bastille console command does not connect me to the console.
Reverting the fstab entry back to 13.3-RELEASE makes the bastille console command work again.

[MANDATORY] Bastille and FreeBSD version (paste bastille -v && freebsd-version -kru output)
$ bastille -v && freebsd-version -kru
0.10.20231125
14.1-RELEASE
14.1-RELEASE
14.1-RELEASE

[MANDATORY] How did you install bastille? (port/pkg/git)

pkg
[optional] Steps to reproduce?

[13:14:39:freebsd:paulz:/var/log/bastille]
$ doas bastille  start unifi
pfctl: Invalid argument.
[unifi]:
unifi: created

[13:14:58:freebsd:paulz:/var/log/bastille]
$ doas bastille  console unifi
[unifi]:

[13:15:04:freebsd:paulz:/var/log/bastille]
$

[optional] Expected behavior
Console prompt for jail console

[optional] Screenshots
If applicable, add screenshots to help explain your problem.

[optional] Additional context

I can start a shell inside the jail using jexec $JID sh.
The unifi service I want to run inside that jail does not seem to be started and when I start it I get the error:

# service unifi start
Starting unifi.
su: pam_start: System error
/usr/local/etc/rc.d/unifi: WARNING: failed to start unifi

[michael-o]

Does a fresh jail work?

[paulz-gh]

Yes a new jail created with bastille -C 14.1-RELEASE works.
I tried one of my other 13.3-RELEASE jails and that also fails. So it's not just the one I tried first.

I also created a new 13.3-RELEASE jail and edited fstab to point to 14.1-RELEASE. That also fails to get a console.

The console log for that jail shows no errors:
First entries are for 13.3 last is for 14.1
$ cat /var/log/bastille/testjail3_console.log
ELF ldconfig path: /lib/casper /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32
Updating motd:.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating /var/run/os-release done.
Starting syslogd.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.

Thu Jun 6 19:10:45 CEST 2024
Stopping cron.
Waiting for PIDS: 57930.
.
Terminated
ELF ldconfig path: /lib/casper /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32
Updating motd:.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating /var/run/os-release done.
Starting syslogd.
Starting cron.

Thu Jun 6 19:10:47 CEST 2024
Stopping cron.
Waiting for PIDS: 58718.
.
Terminated
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32
Updating motd:.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating /var/run/os-release done.
Starting syslogd.
Starting cron.

Thu Jun 6 19:12:02 CEST 2024

[paulz-gh]

I did some testing and I can start a root shell in updated jail but login command fails.
I see following in /var/log/messages:
Jun 7 11:09:51 unifi login[11869]: in openpam_load_module(): no pam_opie.so found
Jun 7 11:09:51 unifi login[11869]: pam_start(): System error
Jun 7 11:12:07 unifi login[11932]: in openpam_load_module(): no pam_opie.so found
Jun 7 11:12:07 unifi login[11932]: pam_start(): System error

Looks like a missing pam module.
I have copied /etc/pam.d/* to the jail /etc/pam.d and now console works.

[michael-o]

As far as I remember pam_opie has ben removed/replaced. Did freebsd-udpate fail here?
See 14.0 RNs:
The one-time password facility OPIE, opie(4), has been removed from the base system. If you still wish to use it, install the security/opie port. Otherwise, make sure to remove or comment out any mention of pam_opie and pam_opieaccess from your PAM policies. etcupdate(8) will normally take care of this for the stock policies. 0aa2700123e2

I don't believe that this is a Bastille issue.

[paulz-gh]

Hi,
Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails.
The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

[michael-o]

Hi, Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails. The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

Consider it incomplete. It is imperative to study official RNs before upgrading...

[paulz-gh]

Hi, Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails. The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

Consider it incomplete. It is imperative to study official RNs before upgrading...

Agreed. But maybe good to add a reminder for that on that page....

[michael-o]

Hi, Maybe I missed something but the procedure on https://bastille.readthedocs.io/en/latest/chapters/upgrading.html#revert-upgrade-downgrade-process does not mention anything about updating the /etc content inside the jails. The jails /etc/pam.d/* references to opie cause the login command in the jail to fail.

Consider it incomplete. It is imperative to study official RNs before upgrading...

Agreed. But maybe good to add a reminder for that on that page....

Given that no issues were addressed for the past 7+ months and the maintainers have disappeared I don't expect this to be fixed anytime soon.

[stafwag]

I had the same issue a few months back.

This might be related to the /etc/ directory isn't updated. So it might be that etcupdated; https://man.freebsd.org/cgi/man.cgi?etcupdate needs to be executed.

There is already a PR open for this: #660

You can still access the "console by' executing jexec.

# jls
# jexec <JID> /bin/sh

I switched from thin jails to tick jails and this also resolves the issues as /etc is updated as part of the tick jail upgrade process.

[yaazkal]

This are the steps to update jails. I think we need to document that here in the official documentation: https://gist.github.com/cedwards/1e00c3d0aa6fbb14bc5b16fca8df0c35