With KeyCloak, After supplying credentials, it just hangs

[rabejens]

I followed the guide, and specified the necessary information. Here is my config:

[default]
region = eu-west-1
saml_auth_url = https://mycompany.com/saml
saml_username = myusername
role_arn = arn:aws:iam::123456789123:role/MyRole

Then I did: aws-runas password and entered the password.

When I now do
aws-runas -v I get output like this:

2020/03/17 08:36:23 DEBUG ENV Config: &{CaBundle: CredentialSource: DurationSeconds:0 ExternalId: MfaSerial: Profile: Region: RoleArn: RoleSessionName: SourceProfile: rawAttributes:map[]}
2020/03/17 08:36:23 DEBUG USER Config: &{CaBundle: CredentialSource: DurationSeconds:0 ExternalId: MfaSerial: Profile:default Region: RoleArn: RoleSessionName: SourceProfile: rawAttributes:map[]}
2020/03/17 08:36:23 DEBUG PROFILE Config: &{CaBundle: CredentialSource: DurationSeconds:0 ExternalId: MfaSerial: Profile:default Region:eu-west-1 RoleArn:arn:aws:iam::123456789123:role/MyRole RoleSessionName: SourceProfile: rawAttributes:map[region:eu-west-1 role_arn:arn:aws:iam::123456789123:role/MyRole saml_auth_url:https://mycompany.com/saml saml_username:myusername]}
2020/03/17 08:36:24 DEBUG MERGED Config: &{CaBundle: CredentialSource: DurationSeconds:0 ExternalId: MfaSerial: Profile:default Region:eu-west-1 RoleArn:arn:aws:iam::123456789123:role/MyRole RoleSessionName: SourceProfile: rawAttributes:map[region:eu-west-1 role_arn:arn:aws:iam::123456789123:role/MyRole saml_auth_url:https://mycompany.com/saml saml_username:myusername]}
2020/03/17 08:36:24 DEBUG FINAL Config: &{AwsConfig:0xc000226960 CredentialsDuration:0s SessionTokenDuration:0s JumpRoleArn:arn::::: SamlAuthUrl:https://mycompany.com/saml SamlUsername:myusername}
2020/03/17 08:36:24 DEBUG Using SAML Identity
2020/03/17 08:36:24 DEBUG found saml_password data: 'XXXXXXX...'

After that, it hangs.

I tried saml2aws which works fine but doesn't support credential_process.

We are using KeyCloak in our company.

What am I missing?

[mmmorris1975]

Thank you for writing up this issue Jens. Is your Keycloak server configured to require MFA, or is simply using the username and password sufficient? Also, do you know which version of the Keycloak server you are using?

After that last line of output, the code is starting to reach out over the network to your Keycloak server. There are a few things which could be happening here -- The first thing it does is send an HTTP HEAD request to your saml_auth_url to figure out which SAML client to use for handling the authentication and SAML actions. If your Keycloak server is slow, or somehow denying this initial HTTP request, I could foresee it hanging at this point. If the HEAD is successful, the next thing it does is attempt to retrieve the SAML response required by AWS, which should be relatively quick. If the previous step fails, the SAML authentication logic kicks in and the tool goes through the effort of authenticating your credentials against the Keycloak server. If MFA is required, you would be prompted for it at this point. After the authentication is complete, it will attempt to get the SAML response needed for AWS again (which again, should be quick, and this time succeed).

Since saml2aws appears to be working for you, it should be safe to assume that all of the networking would be in place to allow you to talk to Keycloak. The only thing that aws-runas could be doing which is different from saml2aws is that HTTP HEAD request to the url.

[mmmorris1975]

As I was researching this, I noticed there were some changes released in Keycloak 8 which I needed to make changes in aws-runas for. The fix only applies to situations where you need to use MFA during authentication to Keycloak, and you are using Keycloak 8 or higher. If that is your setup, please test if aws-runas 2.0.1 addresses your issue, otherwise I can keep investigating if you can provide more details.

[rabejens]

Hello,

we don't use MFA, and I don't know which version of KeyCloak we are using. We have a login portal which is used for multiple services, and that redirects accordingly.

With the new version, it goes one step further:

2020/03/23 10:17:22 DEBUG found saml_password data: 'XXXXXXX...'
2020/03/23 10:17:23 DEBUG divining SAML client

After that, it hangs.

I will point people in charge to this conversation.

[synergiator]

hi there, we use currently KeyCloak 8.0.1 and prepare to rollout 9.0.

[mmmorris1975]

Thank you for the info, and that extra debug data! The update I made for Keycloak 8 really only affects MFA, and the debug statements look like it isn't getting that far down the path in the 1st place.

That final message is at the point where it does the HTTP HEAD request to the saml_auth_url to see if it can find a client based on the response to that request. My guess is that the hang is because it's waiting for a response to the request, and if you wait long enough, you'll eventually get the timeout error from the OS network stack. If you have access to the curl utility, you could try this command:

curl -I https://mycompany.com/saml

and see what happens (using your actual endpoint, of course).

A bit more details about the process here. aws-runas uses IDP-initiated authentication to do the SAML stuff, and the way I found to do that with Keycloak is to have the saml_auth_url setting configured the same as the AWS client URL configured for the IDP-initiated authentication. For example:

http://localhost:8088/auth/realms/master/protocol/saml/clients/aws

is the URL I've configured on my local machine to test Keycloak 8. Looking at the saml2aws example at the end of the Configuring IDP Accounts section, it appears they do the same.

[mmmorris1975]

Looking at the saml2aws code, it seems that they require you to explicitly specify the IDP you are using in the configuration. It may be worth adding similar logic to aws-runas so people have the ability to skip this auto-detection logic if it isn't working for them.

[mmmorris1975]

I may have been able to get some insight in to this, as I saw something similar with my setup (it doesn't use Keycloak, but my hunch is that the issue is not IDP-specific). The CDN that sits in front of the IDP used at my employer was having some kind of issue, causing the auto-detection to timeout and fail sporadically. To get around this I implemented a feature similar to what is used with saml2aws, where you can explicitly set a provider in your configuration to bypass the auto-detection logic.

If you get aws-runas 2.0.2 and add the following line with the rest of your saml configuration, it may address your issue:

saml_provider = keycloak

[rabejens]

This goes one step further before crashing:

2020/03/26 08:54:27 DEBUG Using SAML Identity
2020/03/26 08:54:27 DEBUG found saml_password data: 'XXXXXXXX...'
2020/03/26 08:54:28 DEBUG divining SAML client (keycloak)
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
aws-runas/lib/saml.(*keycloakSamlClient).parseRealm(0xc00008ab70)
        /Users/morrm1/GolandProjects/aws-runas/lib/saml/keycloak_client.go:43 +0x118
aws-runas/lib/saml.NewKeycloakSamlClient(0xc0102aa280, 0x5e, 0xd9e21d, 0x8, 0x0)
        /Users/morrm1/GolandProjects/aws-runas/lib/saml/keycloak_client.go:29 +0xbf
aws-runas/lib/saml.GetClient(0xc000084af1, 0x8, 0xc0102aa280, 0x5e, 0xc0102a3c60, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
        /Users/morrm1/GolandProjects/aws-runas/lib/saml/client_factory.go:29 +0xbe
main.samlClientWithReauth(0xc00006d550, 0x5, 0xc00012bd28, 0x1)
        /Users/morrm1/GolandProjects/aws-runas/main.go:568 +0x2b1
main.awsUser(0xc000222750, 0xd9e3f5)
        /Users/morrm1/GolandProjects/aws-runas/main.go:530 +0x158
main.main()
        /Users/morrm1/GolandProjects/aws-runas/main.go:77 +0x1cd

[mmmorris1975]

First, I want to thank you for your patience and help working through this, it's very helpful for me to see ways others are using this tool outside of my limited testing ability for the various SAML implementations.

Second, this error is 100% my fault, going back over the code where this is failing, it's doing some URL parsing which isn't even used as part of the SAML or authentication process. I should hopefully have a fix (to remove the dead code paths) ready before you start your day again.

[mmmorris1975]

Please try 2.0.3 when you have the time, it should get past this error as that release no longer parses the auth url (since the parsed results were never used anywhere)

[rabejens]

2020/03/30 09:12:25 DEBUG Using SAML Identity
2020/03/30 09:12:25 DEBUG found saml_password data: 'XXXXX...'
2020/03/30 09:12:25 DEBUG divining SAML client (keycloak)
2020/03/30 09:12:25 DEBUG checking SAML response
2020/03/30 09:12:26 DEBUG doing SAML authentication
2020/03/30 09:12:26 FATAL auth status code 401

I double-checked that login URI and password are both correct. I also deleted the credentials file and entered the password at the command line to no avail.

[mmmorris1975]

It's progress, but not the result I was hoping for. I'm trying to compare what my code is doing differently then saml2aws when doing the Keycloak authentication. The only obvious thing is that there's a hidden field which saml2aws is passing, which I wasn't including. In my local setup, that hidden field didn't matter, the authentication still succeeded if it wasn't included in the POST for the authentication, but all I'm doing is running a keycloak docker container (versions 6.0.1, 8.0.1, 8.0.2, and 9.0.2), creating a user, and configuring an AWS SAML client. In a more "production-ready" setup, that field may be important.

My hope is that including these hidden fields from the login form will get the authentication working. Please try version 2.0.4 when you have the chance. Thanks!

[synergiator]

A new test with 2.0.5 get the following results:

...
2020/04/22 10:34:48 DEBUG AssumeRole CACHE PATH: /home/USER/.aws/.aws_assume_role_saml
2020/04/22 10:34:48 cache load error: open /home/pmuryshkin/.aws/.aws_assume_role_saml: no such file or directory
...
2020/04/22 10:34:48 FATAL Error getting credentials: InvalidParameter: 1 validation error(s) found.
- minimum field size of 20, AssumeRoleWithSAMLInput.PrincipalArn.

From the documentation, it's not quite clear where to place the password for SSO/SAML scenario.
Does the tool integrate with the feature "credential_process" of AWS cli?

[mmmorris1975]

Some good news about that error, the authentication with Keycloak was successful, and aws-runas should have retrieved the SAMLResponse document from Keycloak, and is attempting to do the AssumeRoleWithSaml API call. The term "credentials" with that error message is probably a bit mis-leading, it's referencing the AWS role credentials at this point, and not your personal Keycloak credentials.

To find the bit of information the error message is complaining about, it applies a regex to the SAMLResponse document to find the elements which list the roles you have access to, and the associated saml-provider ARN used to integrate the AWS authentication with Keycloak. If you're fluent in regular expressions, it's doing this:

`>(arn:aws:iam::\d+:role/.*?),(arn:aws:iam::\d+:saml-provider/.*?)<`

and is using the role_arn from your configuration to find a match for that first bit in parenthesis, and set the PrincipalArn value from that second part in parenthesis, if a match is found. It appears that the matching failed, but I need to figure out why. It could be the case that I need to adjust the regular expression, it assume the data lives on a single line, and with the testing I've done so far, that has held true, although maybe this is the case where that makes that assumption fail? I've also seen configuration errors in either the role ARN in someone's local configuration, or Keycloak SAML setup cause this. The next part of the investigation to narrow that down requires that you decode the SAMLResponse data that's output by the DEBUG log level, and see what Keycloak is handing to AWS for the SAML authentication. The information we're looking for is contained under the XML tag <saml:Attribute Name="https://aws.amazon.com/SAML/Attributes/Role"> and will provide the set of AWS roles, and their associated AWS integration ARN values.

[mmmorris1975]

To answer the actual question you asked about your personal SAML credentials, the tool will accept your SAML password as an environment variable SAML_PASSWORD, or using the -P command line option. There's also the password sub-command of aws-runas which will place and obfuscated form of you password in the .aws/credentials file. This may be instructive to help with that.

[synergiator]

hi @mmmorris1975, thank you for providing fast responses!

Please provide instructions how I should debug the output/SAML response to share the information you need to fix the problem.

Another question, does the tool integrate with the feature "credential_process" of AWS cli?

[mmmorris1975]

Sorry about that, I should have included some guidance on how to go about that.  In the debug output from the command, copy the block of text following the line DEBUG SAMLResponse: (it will be pretty long, going until you see the next line with DEBUG in it).

On my Mac, I save the SAMLResponse info directly from the aws-runas output in a text file (I'll call it my_file in examples) and from the command line, run

base64 -D < my_file | xmllint --format -

(That last "-" is important!) which will provide the XML payload of the SAMLResponse in a format that makes it easier to look at.  Linux has similar commands, and the only difference I found is that the -D argument you use with the base64 command on a Mac is -d on Linux.
On Windows (although these steps will work for Linux and Mac too, if you don't want to deal with the command line tools), you'll need to paste the SAMLResponse from aws-runas in a website like https://www.freeformatter.com/base64-encoder.html using the Decode button, which will decode the data in to an ugly looking XML document.  To make that XML more readable, you can take that XML information and paste it in the same site at https://www.freeformatter.com/xml-formatter.html.

Once you have the XML document, near the bottom you will see the AWS ARNs of the roles you have access to, and the SAML integration ARN (PrincipalArn) in the <saml:AttributeValue> tags under a parent <saml:Attribute Name="https://aws.amazon.com/SAML/Attributes/Role"> tag.  When aws-runas processes this output, it is doing a case-sensitive exact match for the role ARN from your local configuration to find the other info needed to perform the AssumeRole operation.

It won't give you the full information that the steps above do, but since it seems we're getting a SAMLResponse from Keycloak at this point, you should be able to use the aws-runas -l command, and it will print the role ARNs that it is finding in the data (it doesn't include that PrincipalArn information).  At the very least you can use that to double check your local configuration of the role ARN against what Keycloak is providing to AWS.

[mmmorris1975]

aws-runas doesn't explicitly process any credential_process configuration in a profile, however I've never tested it to see how the AWS SDK, which is underneath all of this, would behave if that is configured.

Using aws-runas as the command for the command_process does work, with some caveats. I talked through this with someone as part of #44

[rabejens]

I just had time to revisit this, and with the latest version, I still get:

2020/08/25 13:32:34 FATAL Error getting credentials: InvalidParameter: 1 validation error(s) found.
- minimum field size of 20, AssumeRoleWithSAMLInput.PrincipalArn.

but when running it with the -l option, I get like:

Available role ARNs for [email protected]
  arn:aws:iam::123456789123:role/TheRoleIWant

So, it finds the roles but somehow doesn't like them.

My .aws/config looks like this:

[default]
region = eu-west-1
saml_auth_url = https://loginsite.com
saml_username = [email protected]
role_arn = arn:aws:iam::123456789123:role/TheRoleIWant
saml_provider = keycloak

[mmmorris1975]

That particular error has something to do with the SAML assertion coming back from KC. The SAML doc has the role and IAM principal ARN (used for the SSO integration), wrapped up in the values in the Role attribute.

There's a chance it's a data condition I'm not handling here, if you're able to provide some details about the SAML response from KC, that could be helpful. The latest version of aws-runas will parse and display the interesting bits for this after the base64 encoded saml document. It will look something like:

2020/09/02 11:30:07 DEBUG SAML Role Details:
  arn:aws:iam::01234567890:saml-provider/keycloak arn:aws:iam::01234567890:role/Admin
  arn:aws:iam::09876543210:saml-provider/keycloak arn:aws:iam::09876543210:role/Admin

You could also open up the encoded saml document and find the values in the https://aws.amazon.com/SAML/Attributes/Role saml:Attribute if you want to see the raw data, of feel like it would be helpful to resolve this issue

[mmmorris1975]

I had someone in my company ask about that same validation error. In their case, it was a typo/misconfiguration in the role ARN in their local config, which caused aws-runas to not match the role when it was examining the SAML assertion, resulting in an empty Principal ARN being used for the STS calls. I'll work on making that more robust (or at least a less cryptic error message), but thought I would pass along that information, in case it's helpful in tracking down the source of the issue.