Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSM agent update disables auto restart of SSM agent service across reboots on SUSE #617

Open
umair110-IT opened this issue Feb 7, 2025 · 3 comments

Comments

@umair110-IT
Copy link

Recently, while performing patching on EC2 instance running SUSE Linux Enterprise Server 15 SP5, we applied all OS security updates. These updates also included amazon-ssm-agent security update released by SUSE on Jan 28th as below;

[SUSE Link] (https://www.suse.com/support/update/announcement/2025/suse-su-20250277-1/)

After applying the OS updates and rebooting the server, the server could not be connected via SSM. After investigation, I found out that the SSM agent service was disabled to start on boot. I enabled the SSM agent service and it started successfully and worked without any issues afterwards.

I tried to reproduce the problem and this time I manually updated SSM agent to see what it was doing. During this new SSM agent update, it actually removed the SSM agent service file from systemd, which is responsible to start service automatically.

Here is how the logs look during SSM agent update:

The following package is going to be upgraded:
  amazon-ssm-agent

The following package is going to change vendor:
  amazon-ssm-agent  Amazon.com -> SUSE LLC <https://www.suse.com/>

Continue? [y/n/v/...? shows all options] (y): y
Retrieving: amazon-ssm-agent-3.3.1611.0-150000.5.20.1.x86_64 (SLE-Module-Public-Cloud15-SP5-Updates)
Checking for file conflicts: ..........................................................................................................................................................................................................[done]
Removed /etc/systemd/system/multi-user.target.wants/amazon-ssm-agent.service.

Affected SSM agent version: amazon-ssm-agent-3.3.1611.0-150000.5.20.1.x86_64

The solution is to manually enable the amazon-ssm-agent service so it can start automatically across reboots. This also recreates the missing service file under systemd.

# systemctl enable amazon-ssm-agent
Created symlink /etc/systemd/system/multi-user.target.wants/amazon-ssm-agent.service → /usr/lib/systemd/system/amazon-ssm-agent.service.

The ownership of this package is by SUSE (as can be seen in update logs), I am following up with them to review this further.

@umair110-IT
Copy link
Author

This is being worked on by SUSE here:
https://bugzilla.opensuse.org/show_bug.cgi?id=1236881

@TLaue
Copy link

TLaue commented Feb 17, 2025

I am facing the same issue. Unfortunately, the bug report is not accessible - even when logged into Bugzilla. Any details about the issue available and even more important any idea when it will get fixed.

Thanks in advance for your help.

@Aperocky
Copy link
Contributor

Aperocky commented Feb 17, 2025

AWS vended version of the agent (acquired from AWS-UpdateSSMAgent etc) have a conflict with SUSE vended package, When running patch a security flag on zypper caused the SUSE package to be loaded instead, the default configuration there have enabled=false, causing the agent to go offline.

If you are:

  1. Running SSM Agent on SUSE Linux.
  2. Have Auto-Update turned on, or have manually installed the agent (not via zypper), or have ran AWS-UpdateSSMAgent document
  3. Scheduled or planning to run patch.

We advise you to add amazon-ssm-agent-3.3.1611.0-150000.5.20.1* to rejected-patches parameter and ensure that rejected-patches-action is set to BLOCK to prevent agent disconnection from happening.

The new version that is currently rolling out will mitigate this issue: https://github.com/aws/amazon-ssm-agent/releases/tag/3.3.1802.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants