Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Argo S3 artifacts don't work when minio client needs to assume role #11723

Closed
3 tasks done
firecast opened this issue Sep 1, 2023 · 3 comments
Closed
3 tasks done

Argo S3 artifacts don't work when minio client needs to assume role #11723

firecast opened this issue Sep 1, 2023 · 3 comments
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc go Pull requests that update Go dependencies type/bug type/dependencies PRs and issues specific to updating dependencies type/regression Regression from previous behavior (a specific type of bug)

Comments

@firecast
Copy link

firecast commented Sep 1, 2023

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Expected:

  • S3 artifacts would be fetched where the IAM role had to assume another role

Happened:

  • Pod wait container was not able to fetch artifacts and gave error
    http://169.254.169.254/latest/api/token": context deadline exceeded

This happens if the EC2 instances is not enabled for IMDSv2 and was introduced as part of this PR on minio-go and was merged in Argo v3.4.0. It has since been fixed and released as a BugFix version v7.0.63. We will need to just update the the minio dependency to fix it.

Version

v3.4.9 and latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: ghost-transform
  namespace: argo
spec:
  podGC:
    strategy: OnPodSuccess
    deleteDelayDuration: 500s
  templates:
    - name: tails
      script:
        image: python:alpine3.6
        command: [ python ]
        source: |
          with open("result.txt", "w") as f:
            f.write("it was tails")
      inputs:
        artifacts:
        - name: input
          path: "/tmp/inputs"
      outputs:
        artifacts:
          - name: result
            path: /result.txt

    - name: main
      dag:
        tasks:
          - name: run
            template: tails
            arguments:
              artifacts:
                - name: input
                  s3: # access to this artifact required the minio client to assume a role
                    bucket: "some-non-default-artifact-bucket"
                    key: "some-prefix"
                    region: "eu-central-1"
                    endpoint: "s3.eu-central-1.amazonaws.com"
                
  entrypoint: main

Logs from the workflow controller

Not relevant

Logs from in your workflow's wait container

http://169.254.169.254/latest/api/token": context deadline exceeded
@firecast firecast added type/bug type/regression Regression from previous behavior (a specific type of bug) labels Sep 1, 2023
@agilgur5 agilgur5 added type/dependencies PRs and issues specific to updating dependencies go Pull requests that update Go dependencies area/artifacts S3/GCP/OSS/Git/HDFS etc labels Sep 1, 2023
@terrytangyuan
Copy link
Member

Would you like to submit a PR to upgrade it?

@agilgur5
Copy link
Contributor

agilgur5 commented Sep 4, 2023

This was updated in argoproj/pkg as part of argoproj/pkg#492, so just pkg needs a bump. May happen as part of #11704 (comment)

@agilgur5
Copy link
Contributor

agilgur5 commented Sep 5, 2023

Should be fixed by #11704 now

@agilgur5 agilgur5 closed this as completed Sep 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc go Pull requests that update Go dependencies type/bug type/dependencies PRs and issues specific to updating dependencies type/regression Regression from previous behavior (a specific type of bug)
Projects
None yet
Development

No branches or pull requests

3 participants