Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adds GCP secret manager triggerauthentication #4864

Merged
merged 41 commits into from
Jan 15, 2024

Conversation

neelanjan00
Copy link
Contributor

@neelanjan00 neelanjan00 commented Aug 6, 2023

A description of what has been changed

  • Adds GCP secret manager as a source for scaler TriggerAuthentication

Checklist

Fixes #4831

Relates to kedacore/keda-docs#1203

@neelanjan00 neelanjan00 requested a review from a team as a code owner August 6, 2023 08:18
@github-actions
Copy link

github-actions bot commented Aug 6, 2023

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

🏖️ Over the summer, the response time will be longer than usual due to maintainers taking time off so please bear with us.

While you are waiting, make sure to:

Learn more about:

@SpiritZhou
Copy link
Contributor

Would you mind adding some e2e tests?

@neelanjan00
Copy link
Contributor Author

Would you mind adding some e2e tests?

Sure, is there any guide to how to get started on this?

@SpiritZhou
Copy link
Contributor

Would you mind adding some e2e tests?

Sure, is there any guide to how to get started on this?

You can read the doc here: https://github.com/kedacore/keda/blob/main/tests/README.md and see examples in tests/secret-provider. Maybe you need to update E2E Test infrastructure to add GCP secret manager also.

@JorTurFer
Copy link
Member

Yeah, please add an e2e test for this feature (which is awesome BTW).
let us know if you need help with something 😄

@neelanjan00
Copy link
Contributor Author

neelanjan00 commented Aug 16, 2023

Yeah, please add an e2e test for this feature (which is awesome BTW). let us know if you need help with something 😄

Hi Jorge, I will be very glad if you can explain the overall e2e test flow. Upon checking the E2E test infrastructure repo, I found Terraform scripts which I presume will provision the requisite resources needed to perform the test. But at which phase of the test are they invoked? Also, I am new to Terraform so it will be helpful if someone can list the steps needed to be followed to use The GCP Secret Manager.

@JorTurFer
Copy link
Member

Hi Jorge, I will be very glad if you can explain the overall e2e test flow. Upon checking the E2E test infrastructure repo, I found Terraform scripts which I presume will provision the requisite resources needed to perform the test. But at which phase of the test are they invoked? Also, I am new to Terraform so it will be helpful if someone can list the steps needed to be followed to use The GCP Secret Manager.

Let's go step by step xD

I found Terraform scripts which I presume will provision the requisite resources needed to perform the test.

Yeah, that's it. We define all the testing infrastructure using that repository with helm.

But at which phase of the test are they invoked?

They are executed on merge on that repo. It means that the resources are always available. For the moment, we don't create and delete them because it can increase significantly the execution time. We scale out/in the clusters during the e2e tests, but that's all.

Also, I am new to Terraform so it will be helpful if someone can list the steps needed to be followed to use The GCP Secret Manager.

As this is a gcp resource, you should create a module inside GCP folder or maybe using one like this: https://registry.terraform.io/modules/GoogleCloudPlatform/secret-manager/google/latest.
Then you need to provide the secrets, for example this is how we are doing it for Azure Key Vault:
https://github.com/kedacore/testing-infrastructure/blob/b81a768b349b53ea12ca24ad27cf6fb7de276681/terraform/main.tf#L117-L139

Other option is instead of already provisioning the secrets in terraform, you could do it as part of the tests, as we do for HashiCorp Vault test case: https://github.com/kedacore/keda/blob/main/tests/secret-providers/hashicorp_vault/hashicorp_vault_test.go

After that, you have to set all the secrets that you need for your e2e tests, adding them here: https://github.com/kedacore/testing-infrastructure/blob/main/terraform/main.tf#L242

I will be very glad if you can explain the overall e2e test flow

You already have it explained here: https://github.com/kedacore/keda/blob/main/tests/README.md
But a quick summary is that On merge and on demand, the tests are executed in a real Kubernetes clusters (AKS). The process is like

  • Install KEDA and cluster scoped deps like pod identities
  • Execute all the e2e tests except those that are inside sequential folder in a random order
  • Execute all the e2e tests inside sequential folder
  • Remove KEDA and all the cluster deps

Each e2e test has to setup all the local deps that it needs, and it has to remove them as clean up

As I said, if you have any specific question or you need help with something, you can ping us. If you prefer, I could try to do the terraform changes this week to spin up the service in GCP and set the secrets in the repo

@neelanjan00
Copy link
Contributor Author

Thanks for the detailed explanation Jorge, it clears up all my doubts. Let me take a stab at it.

@neelanjan00 neelanjan00 force-pushed the gcp-secret-manager-triggerauth branch from 90dc34f to fab71a5 Compare September 24, 2023 13:14
@neelanjan00
Copy link
Contributor Author

E2E test logs for GCP Secret Manager TriggerAuth

  • Create a GCP Secret Manager secret
  • Create Kubernetes resources for PostgreSQL server
  • Create Kubernetes resources for testing, including a TriggerAuthentication CR for GCP Secret Manager, consumed by a ScaledObject CR for PostgreSQL
  • Validate activation, scale-in, and scale-out for the PostgreSQL scaler
  • Delete Kubernetes resources for testing
  • Delete GCP Secret Manager secret
=== RUN   TestPostgreSQLScaler
    gcp_secret_manager_test.go:441: Created secret in GCP Secret Manager.
    helper.go:239: deleting namespace gcp-secret-manager-test-ns
    helper.go:292: waiting for namespace gcp-secret-manager-test-ns deletion
    helper.go:226: Creating namespace - gcp-secret-manager-test-ns
    helper.go:514: Applying template: postgreSQLStatefulSetTemplate
    helper.go:514: Applying template: postgreSQLServiceTemplate
    helper.go:428: Waiting for statefulset replicas to hit target. Statefulset - postgresql, Current  - 0, Target - 1
    helper.go:428: Waiting for statefulset replicas to hit target. Statefulset - postgresql, Current  - 1, Target - 1
    helper.go:180: Waiting for successful execution of command on Pod; Output: CREATE TABLE
        , Error: 
    helper.go:514: Applying template: secretTemplate
    helper.go:514: Applying template: deploymentTemplate
    helper.go:514: Applying template: triggerAuthenticationTemplate
    helper.go:514: Applying template: scaledObjectTemplate
    helper.go:514: Applying template: gcpCredentialsSecretTemplate
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 0, Target - 0
    gcp_secret_manager_test.go:374: --- testing activation ---
    helper.go:514: Applying template: lowLevelRecordsJobTemplate
    helper.go:466: Waiting for some time to ensure deployment replica count doesn't change from 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    helper.go:473: Deployment - gcp-secret-manager-test-deployment, Current  - 0
    gcp_secret_manager_test.go:381: --- testing scale out ---
    helper.go:514: Applying template: insertRecordsJobTemplate
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 0, Target - 2
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 0, Target - 2
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 0, Target - 2
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 1, Target - 2
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 1, Target - 2
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 1, Target - 2
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 2, Target - 2
    gcp_secret_manager_test.go:389: --- testing scale in ---
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 2, Target - 0
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 2, Target - 0
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 1, Target - 0
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 1, Target - 0
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 1, Target - 0
    helper.go:408: Waiting for deployment replicas to hit target. Deployment - gcp-secret-manager-test-deployment, Current  - 0, Target - 0
    helper.go:562: Deleting template: gcpCredentialsSecretTemplate
    helper.go:562: Deleting template: scaledObjectTemplate
    helper.go:562: Deleting template: triggerAuthenticationTemplate
    helper.go:562: Deleting template: deploymentTemplate
    helper.go:562: Deleting template: secretTemplate
    helper.go:562: Deleting template: postgreSQLServiceTemplate
    helper.go:562: Deleting template: postgreSQLStatefulSetTemplate
    helper.go:239: deleting namespace gcp-secret-manager-test-ns
    helper.go:292: waiting for namespace gcp-secret-manager-test-ns deletion
    helper.go:292: waiting for namespace gcp-secret-manager-test-ns deletion
    helper.go:292: waiting for namespace gcp-secret-manager-test-ns deletion
    gcp_secret_manager_test.go:473: Deleted secret from GCP Secret Manager.
--- PASS: TestPostgreSQLScaler (121.45s)
PASS
ok      command-line-arguments  122.000s

@JorTurFer
Copy link
Member

JorTurFer commented Sep 24, 2023

/run-e2e gcp_secret_manager
Update: You can check the progress here

CHANGELOG.md Outdated Show resolved Hide resolved
pkg/scaling/resolver/gcp_secretmanager_handler.go Outdated Show resolved Hide resolved
pkg/scaling/resolver/gcp_secretmanager_handler.go Outdated Show resolved Hide resolved
@zroubalik zroubalik changed the title chore: Adds GCP secret manager triggerauthentication feat: Adds GCP secret manager triggerauthentication Jan 11, 2024
@zroubalik
Copy link
Member

nit: should the PR title say chore? this looks imho more like a feat, this will propagate to commit history iirc.

good catch!

pkg/scaling/resolver/scale_resolvers.go Outdated Show resolved Hide resolved
apis/keda/v1alpha1/triggerauthentication_types.go Outdated Show resolved Hide resolved
pkg/scaling/resolver/gcp_secretmanager_handler.go Outdated Show resolved Hide resolved
@neelanjan00
Copy link
Contributor Author

Thank you so much for the detailed review @wozniakjan, I have added all the requested changes. The conflict has been resolved as well.

@zroubalik
Copy link
Member

zroubalik commented Jan 11, 2024

/run-e2e secret-provider
Update: You can check the progress here

Copy link
Member

@zroubalik zroubalik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, @neelanjan00 the only outstanding nit is the rename to credentials, don't forget to update docs PR as well. Thanks!

apis/keda/v1alpha1/triggerauthentication_types.go Outdated Show resolved Hide resolved
@neelanjan00
Copy link
Contributor Author

I have resolved the comment, @zroubalik @wozniakjan thanks for the review. However, the unit test is failing as the TestNATSJetStreamGetMetrics test is failing. I don't think it is due to my changes, can you PTAL?

@zroubalik
Copy link
Member

@neelanjan00 could you please fix static checks?

Checking section: Unreleased
Error: Section: New is not sorted correctly. Correct order:
- **General**: Adds support for AWS Secret Manager as a source for TriggerAuthentication ([#4628](https://github.com/kedacore/keda/issues/4628))
- **General**: Adds support for GCP Secret Manager as a source for TriggerAuthentication ([#4831](https://github.com/kedacore/keda/issues/4831))
- **General**: Introduce new AWS Authentication ([#4134](https://github.com/kedacore/keda/issues/4134))

Run golangci against the code...............................................Failed
- hook id: golangci-lint
- exit code: 1

pkg/scaling/resolver/scale_resolvers.go:236:1: cyclomatic complexity 35 of func `resolveAuthRef` is high (> 30) (gocyclo)
func resolveAuthRef(ctx context.Context, client client.Client, logger logr.Logger,
^
  1. sort Changelog
  2. add gocyclo ingore to golangci.yml :
    exclude-rules:

Signed-off-by: neelanjan00 <[email protected]>
@neelanjan00
Copy link
Contributor Author

@neelanjan00 could you please fix static checks?

Checking section: Unreleased
Error: Section: New is not sorted correctly. Correct order:
- **General**: Adds support for AWS Secret Manager as a source for TriggerAuthentication ([#4628](https://github.com/kedacore/keda/issues/4628))
- **General**: Adds support for GCP Secret Manager as a source for TriggerAuthentication ([#4831](https://github.com/kedacore/keda/issues/4831))
- **General**: Introduce new AWS Authentication ([#4134](https://github.com/kedacore/keda/issues/4134))

Run golangci against the code...............................................Failed
- hook id: golangci-lint
- exit code: 1

pkg/scaling/resolver/scale_resolvers.go:236:1: cyclomatic complexity 35 of func `resolveAuthRef` is high (> 30) (gocyclo)
func resolveAuthRef(ctx context.Context, client client.Client, logger logr.Logger,
^
  1. sort Changelog
  2. add gocyclo ingore to golangci.yml :
    exclude-rules:

Fixed, PTAL.

@zroubalik
Copy link
Member

zroubalik commented Jan 13, 2024

/run-e2e secret-provider
Update: You can check the progress here

@neelanjan00
Copy link
Contributor Author

Hi @zroubalik / @wozniakjan can you please help with the failing unit tests? TestNATSJetStreamGetMetrics test is failing. I don't think it is due to my changes. We can perhaps merge afterward?

@wozniakjan
Copy link
Member

sure, I will take a look today :)

@wozniakjan
Copy link
Member

indeed a flaky test but it will get better over time - #5372.

either you can rebase on top of the latest main or maybe @zroubalik can you please retrigger the unit tests?

@tomkerkhove
Copy link
Member

tomkerkhove commented Jan 15, 2024

/run-e2e secret-provider
Update: You can check the progress here

@lsteinberg-r7
Copy link

@tomkerkhove @wozniakjan @zroubalik
A whole company is waiting for the merge 😝

@wozniakjan
Copy link
Member

@neelanjan00 given there is no need for a PR to helm chart, can you remove please that item from PR description?

  • A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)

it's holding up the task-list-completed PR check

@wozniakjan
Copy link
Member

now it's just a matter of someone from @kedacore/keda-core-contributors to approve, they will do that momentarily :)

@wozniakjan
Copy link
Member

oh, @neelanjan00, looks like there is a merge conflict again, can you please rebase/merge latest main?

@zroubalik zroubalik merged commit 32df68f into kedacore:main Jan 15, 2024
16 of 18 checks passed
@zroubalik
Copy link
Member

oh, @neelanjan00, looks like there is a merge conflict again, can you please rebase/merge latest main?

I fixed it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support GCP secret manager for scaler Trigger Authentication
7 participants