Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EC2RoleRequestError: no EC2 instance role found after SSM Agent update. #609

Open
DT-NIKOLAR opened this issue Dec 12, 2024 · 2 comments
Open

EC2RoleRequestError: no EC2 instance role found after SSM Agent update. #609

DT-NIKOLAR opened this issue Dec 12, 2024 · 2 comments

Comments

@DT-NIKOLAR
Copy link

We faced an Issue that after an update of the SSM Agent running on an EC2 Instance it returned an error that it cannot find the EC2 Instance Role.
The update is triggered by the SSM Agent Document: AWS-UpdateSSMAgent.
The update was from version: 3.3.1142.0-1.x86_64 to 3.3.1230.0-1.x86_64.
The Issue persisted until the EC2 Instance was restarted.

Stopping amazon-ssm-agent...
2024-11-04 09:07:41.2240 INFO [amazon-ssm-agent] amazon-ssm-agent got signal:terminated value:0x7f30b0e916a0
2024-11-04 09:07:41.2318 INFO [amazon-ssm-agent] Stopping Core Agent
2024-11-04 09:07:41.2422 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Receiving stop signal, stop worker monitor
2024-11-04 09:07:48.2660 INFO [CredentialRefresher] Sending credential refresher stop signal
2024-11-04 09:07:48.2669 INFO [Registrar] Registrar is already stopped
2024-11-04 09:07:48.2696 INFO [amazon-ssm-agent] Bye.
Stopped amazon-ssm-agent.
Updated: amazon-ssm-agent-3.3.1230.0-1.x86_64
Started amazon-ssm-agent.
Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Initializing new seelog logger
New Seelog Logger Creation Complete
2024-11-04 09:08:13.4731 INFO Proxy environment variables:
2024-11-04 09:08:13.4731 INFO https_proxy:
2024-11-04 09:08:13.4732 INFO http_proxy:
2024-11-04 09:08:13.4732 INFO no_proxy:
2024-11-04 09:08:13.4732 INFO Checking if agent identity type OnPrem can be assumed
2024-11-04 09:08:13.4742 INFO Checking if agent identity type EC2 can be assumed
2024-11-04 09:08:13.5829 INFO Agent will take identity from EC2
2024-11-04 09:08:13.5868 INFO [amazon-ssm-agent] amazon-ssm-agent - v3.3.1230.0
2024-11-04 09:08:13.5868 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
2024-11-04 09:08:13.5868 INFO [amazon-ssm-agent] Starting Core Agent
2024-11-04 09:08:13.5868 INFO [amazon-ssm-agent] Registrar detected. Attempting registration
2024-11-04 09:08:13.5869 INFO [Registrar] Starting registrar module
2024-11-04 09:08:13.5900 INFO [EC2Identity] Checking disk for registration info
2024-11-04 09:08:13.5929 INFO [EC2Identity] Registration info found for ec2 instance
2024-11-04 09:08:13.5930 INFO [amazon-ssm-agent] Registration attempted. Resuming core agent startup.
2024-11-04 09:08:13.5931 INFO [CredentialRefresher] credentialRefresher has started
2024-11-04 09:08:13.5931 INFO [CredentialRefresher] Credentials ready
2024-11-04 09:08:13.5933 INFO [CredentialRefresher] Starting credentials refresher loop
2024-11-04 09:08:13.5933 INFO [CredentialRefresher] Next credential rotation will be in 21.183134547233333 minutes
2024-11-04 09:08:14.6429 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker is not running, starting worker process
2024-11-04 09:08:14.6517 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker (pid:28561) started
2024-11-04 09:08:14.6517 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds
Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=systemctl is-active falcon-sensor removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
2024-11-04 10:02:44.9629 WARN EC2RoleProvider Failed to connect to Systems Manager with instance profile role credentials. Err: retrieved credentials failed to report to ssm. Error: EC2RoleRequestError: no EC2 instance role found
2024-11-04 10:02:45.1341 ERROR [TokenRequestService] failed to retrieve instance identity role. Error: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance": dial tcp 169.254.169.254:80: connect: network is unreachable
2024-11-04 10:02:45.1353 ERROR EC2RoleProvider Failed to connect to Systems Manager with SSM role credentials. error calling RequestManagedInstanceRoleToken: unable to build RSA signature. No Authorization header in request
2024-11-04 10:02:45.1353 ERROR [CredentialRefresher] Retrieve credentials produced error: no valid credentials could be retrieved for ec2 identity. Default Host Management Err: error calling RequestManagedInstanceRoleToken: unable to build RSA signature. No Authorization header in
2024-11-04 10:02:45.1353 INFO [CredentialRefresher] Sleeping for 5m0s before retrying retrieve credentials

Output from /etc/os-release

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
@Aperocky
Copy link
Contributor

Are there full logs after:

2024-11-04 09:08:14.6517 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds
Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=systemctl is-active falcon-sensor removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
2024-11-04 10:02:44.9629 WARN EC2RoleProvider Failed to connect to Systems Manager with instance profile role credentials. Err: retrieved credentials failed to report to ssm. Error: EC2RoleRequestError: no EC2 instance role found
2024-11-04 10:02:45.1341 ERROR [TokenRequestService] failed to retrieve instance identity role. Error: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance": dial tcp 169.254.169.254:80: connect: network is unreachable
2024-11-04 10:02:45.1353 ERROR EC2RoleProvider Failed to connect to Systems Manager with SSM role credentials. error calling RequestManagedInstanceRoleToken: unable to build RSA signature. No Authorization header in request
2024-11-04 10:02:45.1353 ERROR [CredentialRefresher] Retrieve credentials produced error: no valid credentials could be retrieved for ec2 identity. Default Host Management Err: error calling RequestManagedInstanceRoleToken: unable to build RSA signature. No Authorization header in
2024-11-04 10:02:45.1353 INFO [CredentialRefresher] Sleeping for 5m0s before retrying retrieve credentials

It looks like the credentials failed to connect to Instance Profile via IMDSv2 and are pending retry in 5 minutes before the log truncated, is that when you restarted the instance? If this error is consistently recorded after, it suggest problem with the IMDS client on the EC2 instance.

@DT-NIKOLAR
Copy link
Author

The only logs from the ssm-agent after this point are the lines listed above. The instance was restarted a few hours after the issue occurred when we noticed we could not connect to the instance using SSM. During the restart, the ssm-agent received a shutdown call and started normally after the restart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Aperocky @DT-NIKOLAR