[question]: accessing pre-signed s3 object urls created by workflow spec

[wesbragagt]

Prior Search

• I have already searched this project's issues to determine if a similar question has already been asked.

What is your question?

I have an Argo workflow that performs some data operations and dumps a file into a private s3 bucket. After that it generates a pre-signed url and publishes as a slack notification for our support team to download the file. Whenever the user lands on the pre-signed url link it shows as a denied "with an explicit deny in an identity-based policy".

After looking through the iam role assumed by this program, I noticed this deny policy:

{
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": [
                        "44.236.185.140",
                        "44.238.164.127",
                        "54.201.123.244",
                        "10.0.192.0/18",
                        "10.0.64.0/18",
                        "10.0.128.0/18",
                        "10.0.2.0/24",
                        "10.0.0.0/24",
                        "10.0.1.0/24"
                    ]
                }
            },
            "Effect": "Deny",
            "NotAction": "s3:*",
            "Resource": "*"
        },
        {
            "Action": "s3:*",
            "Condition": {
                "NotIpAddress": {
                    "aws:VpcSourceIp": [
                        "44.236.185.140",
                        "44.238.164.127",
                        "54.201.123.244",
                        "10.0.192.0/18",
                        "10.0.64.0/18",
                        "10.0.128.0/18",
                        "10.0.2.0/24",
                        "10.0.0.0/24",
                        "10.0.1.0/24"
                    ]
                }
            },
            "Effect": "Deny",
            "Resource": "*"
        }

After I deleted that policy, I was able to download the file.

I understand the intention in design here for blocking traffic outside the cluster. That said I'm looking for some guidance regarding how to enable such flow of exposing a temporary link to download an object that is generated by a workflow for which assumes that role.

What primary components of the stack does this relate to?

terraform

Code of Conduct

• I agree to follow this project's Code of Conduct

[fullykubed]

@wesbragagt Can you clarify where you see this policy? Is it on the bucket or is it an IAM policy?

[wesbragagt]

@fullykubed It's one of the policies associated with the service account related to a wf_spec

[fullykubed]

Ok. Learned something new myself on this one.

Its undocumented, but it appears that S3 presigned urls have the same permissions as the user which created them. Thus if the "user" is the service account which is only allowed to interact with resources from inside the Kubernetes cluster, then the link will also only work from inside the cluster (that is because the URL is actually generated using the credentials of the creator).

This would actually also explain a bug that I have found in our velero implementation that uses presigned urls for logs as well.

We currently do not have the ability to disable the IP blocks b/c we wanted to guarantee that credentials could never be compromised due to a misconfiguration issue. However, we can add support for an optional toggle that reduces the IP blocks for S3 actions specifically because we don't want to prevent you from using valid AWS functionality like presigned urls.

Expect to see that in the next release.

[fullykubed]

Nevermind, the presigned behavior is literally in the first paragraph of the AWS docs.

[wesbragagt]

Yeah, it took me some time debugging to realize the same thing. All good. Glad it's something helpful to the stack as well.

[wesbragagt]

@fullykubed my workaround this was to create an iam user with only the necessary permissions to read objects from the bucket and passing its access/secret key pair as an output secret to the runtime process that generates the pre-signed url.

[fullykubed]

Thanks for the insight. We will also be adding some enhancements in the next release that should make this easier.

[fullykubed]

More enhancements for this are still WIP, but this might be of interest for a more secure way to manage non-IRSA creds: https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_aws_creds