ssm agent fails on occasion during start up of windows.

[NielsGalConclusion]

Hi,

on a daily basis we launch a new windows AMIs with the SSM agent installed. We have quite a lot of services running on the windows machine. We notice that sometimes during startup there is a variety of errors logged on the instance and the SSM service is either gone completely or failed to start. It seems to me like errors that occur during startup will completely fail the SSM service and

The errors are in three categories

1
IMDS is not reachable
2
This media is write protected
3, most recently.
2024-12-05 05:11:15.3972 ERROR [proceedUpdate @ processor.go.517] error while executing install delayer docstate is not loaded

the last few lines of the update logs are the following:

2024-12-05 05:25:32.1818 INFO Initiating move of files from update installation directory
2024-12-05 05:25:32.1996 DEBUG moved file Amazon_SSM_Agent_20241205051343.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.1996 DEBUG moved file Amazon_SSM_Agent_20241205051343_000_AmazonSSMAgentMSI.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.2047 DEBUG moved file Amazon_SSM_Agent_20241205051450.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.2068 DEBUG moved file Amazon_SSM_Agent_20241205051450_000_AmazonSSMAgentMSI.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.2097 DEBUG moved file Amazon_SSM_Agent_20241205051832.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.2102 DEBUG moved file Amazon_SSM_Agent_20241205051832_000_AmazonSSMAgentMSI.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.2102 DEBUG moved file Amazon_SSM_Agent_20241205052215.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.2102 DEBUG moved file Amazon_SSM_Agent_20241205052234.log from C:\ProgramData\Amazon\SSM\Update\installationDir to C:\Windows\TEMP successfully
2024-12-05 05:25:32.4252 INFO initiating cleanup of other versions in amazon-ssm-agent and amazon-ssm-agent-updater folder
2024-12-05 05:25:32.4252 INFO removing artifacts in the folder: C:\ProgramData\Amazon\SSM\Update\amazon-ssm-agent
2024-12-05 05:25:32.4379 INFO removed files and folders: 3.3.1345.0
2024-12-05 05:25:32.4379 INFO removing artifacts in the folder: C:\ProgramData\Amazon\SSM\Update\amazon-ssm-agent-updater
2024-12-05 05:25:32.4448 INFO removed files and folders: 3.3.1230.0
2024-12-05 05:25:32.4448 INFO initiating cleanup of files in update download folder
2024-12-05 05:25:32.4547 INFO Successfully downloaded manifest
Successfully downloaded updater version 3.3.1345.0
Updating amazon-ssm-agent from 3.3.1230.0 to 3.3.1345.0
Successfully downloaded https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-west-1/amazon-ssm-agent/3.3.1230.0/amazon-ssm-agent-windows-amd64.zip
Successfully downloaded https://s3.eu-west-1.amazonaws.com/amazon-ssm-eu-west-1/amazon-ssm-agent/3.3.1345.0/amazon-ssm-agent-windows-amd64.zip
Initiating amazon-ssm-agent update to 3.3.1345.0
failed to update amazon-ssm-agent to 3.3.1345.0, failed to start the agent, ErrorMessage=checking svc and agent worker running err, The specified service does not exist as an installed service.
Initiating rollback amazon-ssm-agent to 3.3.1230.0
failed to rollback amazon-ssm-agent to 3.3.1230.0, failed to start the agent, ErrorMessage=checking svc and agent worker running err, The specified service does not exist as an installed service.
Failed to update amazon-ssm-agent to 3.3.1345.0