Add Podman v5 breaking changes checks. #2818

gursewak1997 · 2024-01-24T00:59:28Z

Podman 5 will come with breaking changes affecting upgradability. CGroups v1 environments will be required to switch to CGroups v2 and CNI plugin environemnts will need to switch to netavark. In the above script we added the CLHM helpers to notifiy people of the incoming changes.
Ref: coreos/fedora-coreos-tracker#1629

jlebon · 2024-01-25T22:21:29Z

Thanks, I think this is a good start. Now, before going further and refining this we need to do two things:

Try to reproduce the situation that triggers this. E.g. start an old enough FCOS then update all the way to e.g. rawhide with podman v5 (it doesn't look like it's in rawhide yet, but you can use the packages from https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/ for now) and confirm the issues.
Find and test the steps to take to remediate the issues. E.g. for cgroups, confirm that just dropping systemd.unified_cgroup_hierarchy=0 and rebooting works (i.e. existing containers still work fine). For the CNI networking case, I think there's no choice but to nuke all the containers, but that needs investigation.

Once we have that info, let's write up a nice comment in e.g. coreos/fedora-coreos-tracker#1629 that we can then link to from these issue dropins.

jlebon · 2024-01-25T22:22:32Z

Also, note we'll need a systemd service to actually run the script.

gursewak1997 · 2024-02-08T09:30:56Z

I see that we have a script coreos-check-cgroups that already checks and warns the users who use cgroupv1.
Do we keep and add networking check to it or delete that.

dustymabe · 2024-02-13T17:20:12Z

overlay.d/14container/usr/libexec/coreos-podman-changes-check

+# Netavark is the default network backend and was added in
+# Podman version 4.0. CNI is deprecated and is removed
+# in Podman version 5.0, in preference of Netavark.
+podmanBackend=$(podman info --format "{{.Host.NetworkBackend}}")


I think we need to somehow not run this check if the users are using docker and not podman. Just running the podman or docker commands on systems that are using the other runtime can cause issues.

@jlebon - do you agree?

Hmm, I'm not sure but I would think podman info at least would be safe regardless. The larger point of only warning if podman is used makes sense, though I'm not sure there's a reliable way to know which runtime is in use. E.g. just checking for running containers and not finding any at that moment in time doesn't necessarily mean it's not in use.

If podman info is safe (/cc @mheon), then implicitly it's checking for the "in use" bit as well since it'll only return something other than the netavark default if the machine has state.

I really think running any podman command does make changes to the system. @mheon would know better than I.

One thing we could try to do is look under certain known directories to see if files have been created there and assume podman is in use if some files exist.

The big issue is usually conflicts in firewall rules, and that only happens if a container is started. podman info on a Docker system should not be a problem.

dustymabe · 2024-02-13T17:22:55Z

I see that we have a script coreos-check-cgroups that already checks and warns the users who use cgroupv1.
Do we keep and add networking check to it or delete that.

I think we should keep the checks separate (i.e. a new check for netavark) BUT we should somehow rename the cgroups check so that if they previously ignored the warning by running systemctl disable coreos-check-cgroups.service then the warning will re-appear.

We should also update the text of the cgroup check to reflect the current state.

jlebon · 2024-02-13T17:44:44Z

overlay.d/14container/usr/libexec/coreos-podman-changes-check

+# Netavark is the default network backend and was added in
+# Podman version 4.0. CNI is deprecated and is removed
+# in Podman version 5.0, in preference of Netavark.
+podmanBackend=$(podman info --format "{{.Host.NetworkBackend}}")


Hmm, I'm not sure but I would think podman info at least would be safe regardless. The larger point of only warning if podman is used makes sense, though I'm not sure there's a reliable way to know which runtime is in use. E.g. just checking for running containers and not finding any at that moment in time doesn't necessarily mean it's not in use.

If podman info is safe (/cc @mheon), then implicitly it's checking for the "in use" bit as well since it'll only return something other than the netavark default if the machine has state.

jlebon · 2024-02-13T18:03:28Z

overlay.d/14container/usr/libexec/coreos-podman-changes-check

+podmanBackend=$(podman info --format "{{.Host.NetworkBackend}}")
+
+if [[ $podmanBackend != "netavark" ]]; then
+ echo -e "${warn}


Instead of just echoing, we need to generate an motd.

jlebon · 2024-02-13T18:06:20Z

overlay.d/14container/usr/libexec/coreos-podman-changes-check

+###########################################################################
+WARNING: This system is using CNI networking. CNI is deprecated and will be
+removed in the upcoming Podman v5.0, in preference of Netavark. To switch
+from CNI networking to netavark, you must run 'podman system reset --force'


Minor: Inconsistent capitalization. Also let's maybe put the command on its own line.

jlebon · 2024-02-13T18:06:32Z

overlay.d/14container/usr/libexec/coreos-podman-changes-check

+WARNING: This system is using CNI networking. CNI is deprecated and will be
+removed in the upcoming Podman v5.0, in preference of Netavark. To switch
+from CNI networking to netavark, you must run 'podman system reset --force'
+command. This will delete all of your images,containers, and custom networks.${nc}"


Suggested change

command. This will delete all of your images,containers, and custom networks.${nc}"

command. This will delete all of your images, containers, and custom networks.${nc}"

gursewak1997 · 2024-02-13T20:20:00Z

We should also update the text of the cgroup check to reflect the current state.

Should we also update the warning to show how to disable this service if needed for eg. add To disable this warning, use: sudo systemctl disable xni-checks.service
Technically they can always disable it but it might not be a good idea to explicitly have in the warning message since we are fully dropping the support for both of them and the transition to cgroupsv2 and netavark is definitely needed.

dustymabe · 2024-02-13T20:25:43Z

We should also update the text of the cgroup check to reflect the current state.

Should we also update the warning to show how to disable this service if needed for eg. add To disable this warning, use: sudo systemctl disable xni-checks.service Technically they can always disable it but it might not be a good idea to explicitly have in the warning message since we are fully dropping the support for both of them and the transition to cgroupsv2 and netavark is definitely needed.

I think it depends on if docker still supports cgroupsv1. If it does then I'd say we should give them an option to disable the message from popping up. If not, then maybe not?

mheon · 2024-02-13T20:33:36Z

overlay.d/14container/usr/libexec/coreos-podman-changes-check

+if [[ $cgroupVersion == "tmpfs" ]]; then
+ echo -e "${warn}
+##########################################################################
+WARNING: This system is using cgroups v1. Podman is dropping support for


Update here: We have formally deprecated CGv1, but are not actually removing until v6.0 (I have no idea when that will come out, but at least a year from now). There are too many stragglers still on v1 (including WSL2), for us to completely remove at this point. So Podman will still work on v1 systems, but will pop up an annoying warning when you do so unless silenced by environment variable.

jlebon · 2024-02-16T19:34:58Z

overlay.d/14container/usr/lib/systemd/system/coreos-cni-networking-check.service

+# CNI networking. If so, they will be warned to move
+# their nodes to netavark respectively.
+[Unit]
+Description=Check if nodes are still using CNI networking


Minor: this should be Title Case

jlebon · 2024-02-16T19:35:45Z

overlay.d/14container/usr/libexec/coreos-cni-networking-check

+podmanBackend=$(podman info --format "{{.Host.NetworkBackend}}")
+
+if [[ $podmanBackend != "netavark" ]]; then
+ motd_path=/run/motd.d/35_cni_warning.motd


Minor: weird indentation level here. Let's do 2 or 4?

jlebon · 2024-02-16T19:36:18Z

overlay.d/14container/usr/libexec/coreos-cni-networking-check

+WARNING: This system is using CNI networking. CNI is deprecated and will be
+removed in the upcoming Podman v5.0, in preference of netavark. To switch
+from CNI networking to netavark, you must run 'podman system reset --force'
+command. This will delete all of your images, containers, and custom networks.


Missing trailing ### divider.

Maybe we should mention that "depending on your setup it may be preferable to reprovision the whole machine from the latest images". because reprovisioning will set things up the right way too.

jlebon · 2024-02-16T19:39:32Z

overlay.d/14container/usr/libexec/coreos-cni-networking-check

+removed in the upcoming Podman v5.0, in preference of netavark. To switch
+from CNI networking to netavark, you must run 'podman system reset --force'


Minor: Netavark (I think? Based on upstream docs, it seems like it's capitalized when talking about the project.)

will podman system reset --force delete your existing containers? If so, maybe we should have a FAQ entry in our docs we should point people to instead for more context so they can make the right decisions.

ok I see line 27 below now. :(

jlebon · 2024-02-16T19:47:08Z

overlay.d/14container/usr/libexec/coreos-cni-networking-check

+ cat << EOF > "${motd_path}"
+${warn}
+###########################################################################
+WARNING: This system is using CNI networking. CNI is deprecated and will be


Suggested change

WARNING: This system is using CNI networking. CNI is deprecated and will be

WARNING: Podman is using CNI networking. CNI is deprecated and will be

jlebon · 2024-02-16T20:00:12Z

overlay.d/14container/usr/libexec/coreos-cni-networking-check

+from CNI networking to netavark, you must run 'podman system reset --force'
+command. This will delete all of your images, containers, and custom networks.


Re. whether to show instructions to disable the warning, I think we probably always should provide them. Users should always be free to disregard warnings if they know why it doesn't apply to them.

jlebon · 2024-02-16T20:02:33Z

overlay.d/15fcos/usr/libexec/coreos-check-cgroups-version

-to use cgroups v2, see:
+##########################################################################
+WARNING: This system is using cgroups v1. Podman has dropped support for
+cgroups v1. Move your nodes to cgroups v2 if not already. For instructions


This needs to be adjusted given #2818 (comment). We could delay warning it until when Podman v6.0 arrives, but meh probably good to give an early heads up.

I think also we shouldn't lose the point about it benefiting the whole system, not just Podman.

OK, how about something like this:

This system is using cgroups v1. For increased reliability
it is strongly recommended to migrate this system and your workloads
to use cgroups v2. A future version of Podman will also drop support
for cgroups v1. For instructions on how to adjust kernel arguments
to use cgroups v2, see:

jlebon · 2024-02-16T20:04:21Z

manifests/fedora-coreos.yaml

@@ -36,6 +36,7 @@ conditional-include:

 ostree-layers:
  - overlay/15fcos
+  - overlay/14container


I would call this overlay something clearer, e.g. 14podman-v5. And add a README to document it. Also, let's make it conditional on Fedora 39? It doesn't hurt on Fedora 40 (we just will never warn since podman can't support anything other than Netavark), but it makes it clearer when we can drop it.

just confirming, we don't really need it to be conditional right? Aiming to keep it and update the README.

We don't need it to be conditional, but we also don't need it to be in Fedora 40. What I mean is making it conditional will make it more likely to get cleaned up when we scan all the overlays we can drop when we rebase on top of f40.

dustymabe

I don't see anything alarming. I added some comments.

I'll trust @jlebon to review from here on out.

jlebon

Some superficial things, but LGTM overall. Thanks!

jlebon · 2024-02-21T20:39:40Z

overlay.d/14podman-v5/usr/libexec/coreos-cni-networking-check

@@ -0,0 +1,36 @@
+#!/usr/bin/bash


Missing set -euo pipefail

jlebon · 2024-02-21T20:41:05Z

overlay.d/14podman-v5/usr/libexec/coreos-cni-networking-check

+WARNING: Podman is using CNI networking. CNI is deprecated and will be
+removed in the upcoming Podman v5.0, in preference of Netavark. To switch
+from CNI networking to Netavark, you must run 'podman system reset --force'
+command. This will delete all of your images, containers, and custom networks.


Hmm, "command." looks weird here. Either e.g. "you must run the command '...'." or "you must run '...'.".

jlebon · 2024-02-21T20:42:24Z

overlay.d/15fcos/usr/libexec/coreos-check-cgroups-version

-sudo systemctl disable coreos-check-cgroups.service
-############################################################################
+sudo systemctl disable coreos-check-cgroups-version.service
+###########################################################################


Super minor: this is longer than the first divider.

jlebon · 2024-02-21T20:44:04Z

overlay.d/14podman-v5/usr/lib/systemd/system/coreos-cni-networking-check.service

+# CNI networking. If so, they will be warned to move
+# their nodes to netavark respectively.
+[Unit]
+Description=Check if Nodes Are Still Using CNI Networking


Suggested change

Description=Check if Nodes Are Still Using CNI Networking

Description=Check If Podman Is Still Using CNI Networking

Podman 5 will come with breaking changes affecting upgradability. CGroups v1 environments will be required to switch to CGroups v2 and CNI plugin environemnts will need to switch to netavark. Updated the existing cgroups-version check and added the check for CNI networking

gursewak1997 force-pushed the clhm-podman5 branch 2 times, most recently from d3bc418 to 82adb54 Compare January 24, 2024 01:28

gursewak1997 requested a review from jlebon January 24, 2024 20:50

gursewak1997 force-pushed the clhm-podman5 branch from 82adb54 to 769957c Compare January 31, 2024 22:23

gursewak1997 force-pushed the clhm-podman5 branch 3 times, most recently from 53861f2 to 675c504 Compare February 8, 2024 09:26

gursewak1997 force-pushed the clhm-podman5 branch 2 times, most recently from a779112 to c7f2ae7 Compare February 9, 2024 20:43

gursewak1997 marked this pull request as ready for review February 9, 2024 21:40

dustymabe reviewed Feb 13, 2024

View reviewed changes

jlebon reviewed Feb 13, 2024

View reviewed changes

mheon reviewed Feb 13, 2024

View reviewed changes

gursewak1997 force-pushed the clhm-podman5 branch 8 times, most recently from fa5e27c to 9026b5b Compare February 14, 2024 17:45

gursewak1997 requested a review from dustymabe February 14, 2024 19:50

gursewak1997 requested a review from jlebon February 15, 2024 19:33

jlebon reviewed Feb 16, 2024

View reviewed changes

dustymabe previously approved these changes Feb 16, 2024

View reviewed changes

gursewak1997 dismissed dustymabe’s stale review via c34ec2b February 16, 2024 21:51

gursewak1997 force-pushed the clhm-podman5 branch from 9026b5b to c34ec2b Compare February 16, 2024 21:51

gursewak1997 requested a review from jlebon February 20, 2024 18:26

jlebon previously approved these changes Feb 21, 2024

View reviewed changes

gursewak1997 dismissed jlebon’s stale review via 25b962a February 21, 2024 21:11

gursewak1997 force-pushed the clhm-podman5 branch from c34ec2b to 25b962a Compare February 21, 2024 21:11

jlebon approved these changes Feb 21, 2024

View reviewed changes

jlebon enabled auto-merge (rebase) February 21, 2024 21:13

gursewak1997 disabled auto-merge February 22, 2024 03:43

gursewak1997 enabled auto-merge (rebase) February 22, 2024 03:59

gursewak1997 merged commit 3668ce5 into coreos:testing-devel Feb 22, 2024
2 of 3 checks passed

gursewak1997 deleted the clhm-podman5 branch February 22, 2024 06:48

gursewak1997 mentioned this pull request Feb 22, 2024

Fedora 40 Changes: Podman v5 coreos/fedora-coreos-tracker#1629

Closed

dustymabe mentioned this pull request Nov 15, 2024

systemd refuses to boot with cgroups v1 enabled in v256+ (F41+) coreos/fedora-coreos-tracker#1833

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Podman v5 breaking changes checks. #2818

Add Podman v5 breaking changes checks. #2818

gursewak1997 commented Jan 24, 2024 •

edited

Loading

jlebon commented Jan 25, 2024

jlebon commented Jan 25, 2024

gursewak1997 commented Feb 8, 2024

dustymabe Feb 13, 2024

jlebon Feb 13, 2024

dustymabe Feb 13, 2024

mheon Feb 13, 2024

dustymabe commented Feb 13, 2024

jlebon Feb 13, 2024

jlebon Feb 13, 2024

jlebon Feb 13, 2024

jlebon Feb 13, 2024

gursewak1997 commented Feb 13, 2024

dustymabe commented Feb 13, 2024

mheon Feb 13, 2024

jlebon Feb 16, 2024

jlebon Feb 16, 2024

jlebon Feb 16, 2024

dustymabe Feb 16, 2024

jlebon Feb 16, 2024

dustymabe Feb 16, 2024

dustymabe Feb 16, 2024

jlebon Feb 16, 2024

jlebon Feb 16, 2024

jlebon Feb 16, 2024

jlebon Feb 16, 2024

gursewak1997 Feb 16, 2024

jlebon Feb 16, 2024

dustymabe left a comment

jlebon left a comment

jlebon Feb 21, 2024

jlebon Feb 21, 2024

jlebon Feb 21, 2024

jlebon Feb 21, 2024

	command. This will delete all of your images,containers, and custom networks.${nc}"
	command. This will delete all of your images, containers, and custom networks.${nc}"

		removed in the upcoming Podman v5.0, in preference of netavark. To switch
		from CNI networking to netavark, you must run 'podman system reset --force'

	WARNING: This system is using CNI networking. CNI is deprecated and will be
	WARNING: Podman is using CNI networking. CNI is deprecated and will be

	Description=Check if Nodes Are Still Using CNI Networking
	Description=Check If Podman Is Still Using CNI Networking

Add Podman v5 breaking changes checks. #2818

Add Podman v5 breaking changes checks. #2818

Conversation

gursewak1997 commented Jan 24, 2024 • edited Loading

jlebon commented Jan 25, 2024

jlebon commented Jan 25, 2024

gursewak1997 commented Feb 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustymabe commented Feb 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gursewak1997 commented Feb 13, 2024

dustymabe commented Feb 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustymabe left a comment

Choose a reason for hiding this comment

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gursewak1997 commented Jan 24, 2024 •

edited

Loading