-
Notifications
You must be signed in to change notification settings - Fork 83
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Use --profile instead of AWS_PROFILE for kubeconfig (#1484)
This PR changes our kubeconfig logic to use a `--profile` arg instead of an `AWS_PROFILE` environment variable so it will always use the expected profile. It also parallelizes the relevant tests and simplifies workflows slightly. As a user, if I generate a kubeconfig for a particular profile I would expect that configuration to _always_ use the profile I specified. However, because we rely on `AWS_PROFILE` it is possible for our generated kubeconfig to be inadvertently overridden by the presence of `AWS_ACCESS_KEY_ID`. > Credentials from environment variables have precedence over credentials from the shared credentials and AWS CLI config file. Credentials specified in the shared credentials file have precedence over credentials in the AWS CLI config file. If AWS_PROFILE environment variable is set and the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, then the credentials provided by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will override the credentials located in the profile provided by AWS_PROFILE. https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#id1 I'll note that I discovered this as part of the prep work for pulumi/ci-mgmt#1142. In particular, our tests currently do a few things to exercise profile switching behavior: 1. During CI setup, we set some fixed access keys for the `default` profile. (This is unnecessary.) 2. During CI setup, we set some fixed access keys for an `alt` profile. This is the profile we expect to use in `TestAccAwsProfile*` tests. 3. During `TestAccAwsProfile*` tests we unset `AWS_SECRET_ACCESS_KEY`, `AWS_ACCESS_KEY_ID`, and `AWS_SESSION_TOKEN` for our process. Importantly, (3) is currently implemented such that (a) it prevents parallelization, and (b) subsequent queries to the k8s API server also lack ambient credentials. After I refactored (3) to allow parallelization the tests started failing. Eventually I realized this was because I was unsetting credentials for the `pulumi` subprocess and our test's k8s client now had ambient credentials taking priority over the expected profile.
- Loading branch information
Showing
16 changed files
with
2,274 additions
and
176 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,7 +16,6 @@ on: | |
| env: | ||
| ALT_AWS_ACCESS_KEY_ID: ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} | ||
| ALT_AWS_SECRET_ACCESS_KEY: ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} | ||
| ALT_AWS_PROFILE: ${{ secrets.ALT_AWS_PROFILE }} | ||
| AWS_REGION: us-west-2 | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} | ||
|
|
@@ -209,12 +208,6 @@ jobs: | |
| role-duration-seconds: 7200 | ||
| role-session-name: ${{ env.PROVIDER }}@githubActions | ||
| role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }} | ||
| - name: Configure AWS CLI | ||
| run: | | ||
| aws configure set default.aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| aws configure set default.aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| aws configure set aws_access_key_id ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| aws configure set aws_secret_access_key ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| - name: Clear GitHub Actions Ubuntu runner disk space | ||
| uses: jlumbroso/[email protected] | ||
| with: | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,7 +7,6 @@ on: | |
| env: | ||
| ALT_AWS_ACCESS_KEY_ID: ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} | ||
| ALT_AWS_SECRET_ACCESS_KEY: ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} | ||
| ALT_AWS_PROFILE: ${{ secrets.ALT_AWS_PROFILE }} | ||
| AWS_REGION: us-west-2 | ||
| GITHUB_TOKEN: ${{ secrets.PULUMI_BOT_TOKEN }} | ||
| GOLANGCI_LINT_VERSION: v1.61.0 | ||
|
|
@@ -298,12 +297,6 @@ jobs: | |
| role-duration-seconds: 7200 | ||
| role-session-name: ${{ env.PROVIDER }}@githubActions | ||
| role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }} | ||
| - name: Configure AWS CLI | ||
| run: | | ||
| aws configure set default.aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| aws configure set default.aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| aws configure set aws_access_key_id ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| aws configure set aws_secret_access_key ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| - name: Clear GitHub Actions Ubuntu runner disk space | ||
| uses: jlumbroso/[email protected] | ||
| with: | ||
|
|
@@ -400,12 +393,6 @@ jobs: | |
| role-duration-seconds: 7200 | ||
| role-session-name: ${{ env.PROVIDER }}@githubActions | ||
| role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }} | ||
| - name: Configure AWS CLI | ||
| run: | | ||
| aws configure set default.aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| aws configure set default.aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| aws configure set aws_access_key_id ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| aws configure set aws_secret_access_key ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| - name: Link nodejs binary for testing | ||
| run: | | ||
| cd ${{ github.workspace }}/bin | ||
|
|
@@ -512,12 +499,6 @@ jobs: | |
| role-duration-seconds: 7200 | ||
| role-session-name: ${{ env.PROVIDER }}@githubActions | ||
| role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }} | ||
| - name: Configure AWS CLI | ||
| run: | | ||
| aws configure set default.aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| aws configure set default.aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| aws configure set aws_access_key_id ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| aws configure set aws_secret_access_key ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| - name: Link nodejs binary for testing | ||
| run: | | ||
| cd ${{ github.workspace }}/bin | ||
|
|
@@ -625,12 +606,6 @@ jobs: | |
| role-duration-seconds: 7200 | ||
| role-session-name: ${{ env.PROVIDER }}@githubActions | ||
| role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }} | ||
| - name: Configure AWS CLI | ||
| run: | | ||
| aws configure set default.aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| aws configure set default.aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| aws configure set aws_access_key_id ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| aws configure set aws_secret_access_key ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| - name: Link nodejs binary for testing | ||
| run: | | ||
| cd ${{ github.workspace }}/bin | ||
|
|
@@ -721,12 +696,6 @@ jobs: | |
| role-duration-seconds: 7200 | ||
| role-session-name: ${{ env.PROVIDER }}@githubActions | ||
| role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }} | ||
| - name: Configure AWS CLI | ||
| run: | | ||
| aws configure set default.aws_access_key_id ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| aws configure set default.aws_secret_access_key ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| aws configure set aws_access_key_id ${{ secrets.ALT_AWS_ACCESS_KEY_ID }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| aws configure set aws_secret_access_key ${{ secrets.ALT_AWS_SECRET_ACCESS_KEY }} --profile ${{ secrets.ALT_AWS_PROFILE }} | ||
| - name: Link nodejs binary for testing | ||
| run: | | ||
| cd ${{ github.workspace }}/bin | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,3 @@ | ||
| pulumi>=3.0.0,<4.0.0 | ||
| pulumi-aws>=6.0.0,<7.0.0 | ||
| pulumi-eks>=2.0.0,<3.0.0 | ||
| pulumi-eks>=3.0.0,<4.0.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,4 @@ | ||
| # examples/cluster | ||
|
|
||
| Creates two EKS clusters in the default VPC with two t3.medium nodes. | ||
| - One cluster uses the default configuration. | ||
| - One cluster uses a non-default configuration. | ||
| Creates an EKS cluster in the default VPC with two t3.medium nodes, using a | ||
| non-default AWS profile for credentials. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,42 +1,59 @@ | ||
| import * as aws from "@pulumi/aws"; | ||
| import * as eks from "@pulumi/eks"; | ||
| import * as k8s from "@pulumi/kubernetes"; | ||
| import * as pulumi from "@pulumi/pulumi"; | ||
| import * as process from "process"; | ||
|
|
||
| const projectName = pulumi.getProject(); | ||
|
|
||
| // For CI testing only: used to set profileName to alternate AWS_PROFILE envvar. | ||
| if (!process.env.ALT_AWS_PROFILE) { | ||
| throw new Error("ALT_AWS_PROFILE must be set"); | ||
| } | ||
|
|
||
| if (!process.env.AWS_REGION) { | ||
| throw new Error("AWS_REGION must be set"); | ||
| throw new Error("AWS_REGION must be set"); | ||
| } | ||
|
|
||
| // AWS named profile to use. | ||
| const profileName = process.env.ALT_AWS_PROFILE; | ||
| const profileName = "aws-profile-node"; | ||
| // AWS region to use. | ||
| const region = pulumi.output(process.env.AWS_REGION as aws.types.enums.Region); | ||
|
|
||
| // Create an AWS provider instance using the named profile creds | ||
| // and current region. | ||
| const awsProvider = new aws.Provider("aws-provider", { | ||
| profile: profileName, | ||
| region: region, | ||
| profile: profileName, | ||
| region: region, | ||
| }); | ||
|
|
||
| // Define the AWS provider credential opts to configure the cluster's | ||
| // kubeconfig auth. | ||
| const kubeconfigOpts: eks.KubeconfigOptions = {profileName: profileName}; | ||
| const kubeconfigOpts: eks.KubeconfigOptions = { profileName: profileName }; | ||
|
|
||
| // Create the cluster using the AWS provider and credential opts. | ||
| const cluster = new eks.Cluster(`${projectName}`, { | ||
| const cluster = new eks.Cluster( | ||
| `${projectName}`, | ||
| { | ||
| providerCredentialOpts: kubeconfigOpts, | ||
| }, {provider: awsProvider}); | ||
| corednsAddonOptions: { enabled: false }, // Speed up the test. | ||
| }, | ||
| { provider: awsProvider } | ||
| ); | ||
|
|
||
| // Export the cluster kubeconfig. | ||
| export const kubeconfig = cluster.kubeconfig; | ||
|
|
||
| // Export the cluster kubeconfig with the AWS_PROFILE set. | ||
| export const kubeconfigWithProfile = cluster.getKubeconfig({profileName: profileName}).result; | ||
| export const kubeconfigWithProfile = cluster.getKubeconfig({ | ||
| profileName: profileName, | ||
| }).result; | ||
|
|
||
| const k8sProvider = new k8s.Provider("with-kubeconfig", { | ||
| kubeconfig: kubeconfigWithProfile, | ||
| }); | ||
|
|
||
| // Deploy something into the cluster so upgrade tests can check for unexpected | ||
| // replacements. | ||
| new k8s.core.v1.ConfigMap( | ||
| "cm", | ||
| { | ||
| data: { foo: "bar" }, | ||
| }, | ||
| { provider: k8sProvider } | ||
| ); |
Oops, something went wrong.