PES-3645 Soda <> Redshift IAM Role profiling

[Divyanshu-Patel]

Change Summary

Adding support for Soda profiling w Redshift iam role.
We're now sending a few more attributes to the soda-core in case of Iam role setup.
Why? please check the RCA below.

Packages PR: https://github.com/atlanhq/marketplace-packages/pull/12019

Initial Problem

we were getting the error from the boto library stating partial creds are provided

botocore.exceptions.PartialCredentialsError: Partial credentials found in explicit, missing: aws_access_key_id
reason: somehow, the secret (password) was saved in the creds and the access key wasn’t there - hence, the partial creds error

to fix this, in the sodaConnectionTemplate, we updated the logic to ensure that with IAM role only ARN is sent to the boto library, allowing it to assume the role correctly

Problems while assuming the role

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::015006955552:assumed-role/mercuryins-nodeinstance-role/i-09cee7739ceea8335 
is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::725205928352:role/mig-pr-ebi-atlan-external-role

reason: we are supposed to send the ARN as well as the ExternalId to boto so the role can be assumed (if External_Id is used by the user)

Problems during the get_cluster_credentials call

| botocore.exceptions.ParamValidationError: Parameter validation failed:
| Invalid type for parameter DbUser, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
reason: get_cluster_credentials expected a dbuser as well dbname for the request - we were passing null to it

reason: get_cluster_credentials expected a dbuser as well dbname for the request - we were passing null to it
& because we were not passing it initially, we had to update the sodaConnectionTemplate to use these attributes (we already ask for these details in the Redshift crawler setup)

this still failed stating a policy issue

| botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetClusterCredentials operation: 
User: arn:aws:sts::725205928352:assumed-role/mig-pr-ebi-atlan-external-role/soda_redshift_get_cluster_credentials is not authorized to perform: redshift:GetClusterCredentials
on resource: arn:aws:redshift:eu-west-1:725205928352:dbuser:mercury/atlan_user because no identity-based policy allows the redshift:GetClusterCredentials action

reason: if we look closely, we’re passing the region as eu-west-1 (which is the default region) and we’re seeing the cluster_name as mercury (which we're getting after resolving the host name)
to fix this, pass the cluster id as well as ask for the cluster region in the Redshift config

Jira Issues Resolved

https://atlanhq.atlassian.net/browse/PES-3645