feat: Add options to specify AWS credential source for RDS functions

[hairyhum]

Change Overview

Currently AWS credentials are always taken from the profile, which limits
the ability to export into non-s3 storage and s3 storage in other regions.

Added options to specify a separate secret or use IAM role for service account.

Pull request type

Please check the type of change your PR introduces:

• 🚧 Work in Progress
• 🌈 Refactoring (no functional changes, no api changes)
• 🐹 Trivial/Minor
• 🐛 Bugfix
• 🌻 Feature
• 🗺️ Documentation
• 🤖 Test
• 🏗️ Build

Issues

• fixes #issue-number

Test Plan

• Run RDS example with new arguments added in the blueprint

  • Secret credentials with S3 in another region
  • Serviceaccount credentials

• 💪 Manual

• ⚡ Unit test

• 💚 E2E

[viveksinghggits]

Hi @hairyhum,
I was looking into the changes and was wondering if it would be a good idea to just extend profile in such a way that we can accept the new fields/IAM from profile instead of introducing the new secret way.
I think introducing the secret to accept the creds complicates the things and maybe complicates blueprints as well. And the other reason would be, profile usually is the resource where we say object store details should be defined.
Or, do you think it's not a good idea to just extend the profile for this new thing.

[hairyhum]

I was looking into the changes and was wondering if it would be a good idea to just extend profile in such a way that we can accept the new fields/IAM from profile instead of introducing the new secret way.

There is a principal issue with using profile as a source of the credentials. Because the credentials here are the data source credentials, while profile is by design the data destination credential.
So we either need to extend profile to have both source and destination, which doesn't make sense outside of RDS case, or we need to use another API to supply source credentials.

And the other reason would be, profile usually is the resource where we say object store details should be defined.

And it is still the case even with this change (unless I messed up and introduced a bug). There is no intention to switch away from profiles as destination (e.g. object store) configuration.

[viveksinghggits]

Hi @hairyhum ,
Thank you for the explanation, it looks like I totally misunderstood the PR, sorry about it. I will look into the PR again.

[viveksinghggits]

LGTM overall. left some small comments.

[viveksinghggits]

would it not be a good idea to have another field (credentialsSA) like credentialsSecret, that can can be used to configured the service account?

[hairyhum]

Maybe. The idea here was that an AWS IAM admin would allow kanister service account access to RDS by creating a role for it.
Associating a different service account would be better because it's not trivial to assign multiple roles to a single SA, but implementation-wise it's harder because we currently run RDS operation from within kanister controller (in the function code) meaning that we can't change SA on the flight there.

We can also try using AssumeRole and configure which role it will assume. This is going to be more flexible with less code changes.

I'd suggest we keep it for the follow-up PRs.

[viveksinghggits]

ok 👍 .

[hairyhum]

Did manual testing on RDS. All seems to be working fine.

Will need to update the example readme with a secret source example to address specifics.