Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

env http_proxy in wait/init container does not work for oss client #12313

Closed
2 of 3 tasks
jingkkkkai opened this issue Dec 2, 2023 · 2 comments · Fixed by #12383
Closed
2 of 3 tasks

env http_proxy in wait/init container does not work for oss client #12313

jingkkkkai opened this issue Dec 2, 2023 · 2 comments · Fixed by #12383
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important type/bug

Comments

@jingkkkkai
Copy link
Contributor

jingkkkkai commented Dec 2, 2023

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

in my workflow-controller-configmap had configure http_proxy https_proxy
and it can access gcs s3 artifact repository correctly, except for oss artifact repository

I created a test pod with setting the same proxy, I found that the oss-cli cannot not access the oss either
but I can access the oss bucket after applying the args --proxy-host

# use oss-cli to list buckets 
[root@test /]# aliyun oss ls
^C

# curl 
[root@test /]# curl -I http://${my-oss-bucket}.aliyuncs.com
HTTP/1.1 200 OK
Server: AliyunOSS
Date: Mon, 27 Nov 2023 08:13:57 GMT
Content-Length: 0
x-oss-request-id: xxx
x-oss-bucket-region: xxx
x-oss-server-time: 11
X-Cache: MISS from mail101
X-Cache-Lookup: HIT from mail101:3128
Via: 1.1 mail101 (squid)
Connection: keep-alive

# use oss-cli to list buckets with setting proxy args
[root@test /]# aliyun oss ls oss://${my-oss-bucket} --proxy-host ${my_proxy}
LastModifiedTime                   Size(B)  StorageClass   ETAG                                  ObjectName
2023-11-21 10:00:11 +0000 UTC          130      Standard   XXX      ${my-oss-bucket}/test.tgz
Object Number is: 1

Ref: aliyun http proxy

The oss client in the argo code seems in the same case. The proxy can only be set through oss.ClientOption
However, we cannot configure the proxy for oss client through pod env or argoexec args currently

func (ossDriver *ArtifactDriver) newOSSClient() (*oss.Client, error) {
	var options []oss.ClientOption
        options = append(options, oss.Proxy("${my_proxy}")) <--- here
	if token := ossDriver.SecurityToken; token != "" {
		options = append(options, oss.SecurityToken(token))
	}
	client, err := oss.New(ossDriver.Endpoint, ossDriver.AccessKey, ossDriver.SecretKey, options...)
	if err != nil {
		return nil, fmt.Errorf("failed to create new OSS client: %w", err)
	}
	return client, err
}

Version

v3.4.8 & v3.4.14

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

N/A

Logs from the workflow controller

N/A

Logs from in your workflow's wait container

time="2023-12-02T11:58:27 UTC" level=info msg="Save artifact" artifactName=main-logs duration=37.278144ms error="<nil>" key=2023/12/02/examples.use-cloud-acc-putb
j8br/examples.use-cloud-acc-putbj8br-3396519514/main.log
time="2023-12-02T11:58:27 UTC" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2023-12-02T11:58:27 UTC" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2023-12-02T11:58:27 UTC" level=info msg="Alloc=11282 TotalAlloc=17790 Sys=33661 NumGC=5 Goroutines=7"
time="2023-12-02T11:58:27 UTC" level=fatal msg="timed out waiting for the condition: Put \"http://${my-oss-bucket}.aliyuncs.com/aaa.tgz\": dia
l tcp 47.75.18.199:80: i/o timeout"
@terrytangyuan
Copy link
Member

Would you like to submit a PR?

@agilgur5 agilgur5 added area/artifacts S3/GCP/OSS/Git/HDFS etc P3 Low priority labels Dec 2, 2023
jingkkkkai added a commit to jingkkkkai/argo-workflows that referenced this issue Dec 18, 2023
@jingkkkkai
Copy link
Contributor Author

Would you like to submit a PR?

I opened a PR, can you help me review it?
#12383

@agilgur5 agilgur5 added P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important and removed P3 Low priority labels Jul 2, 2024
jingkkkkai added a commit to jingkkkkai/argo-workflows that referenced this issue Aug 20, 2024
jingkkkkai added a commit to jingkkkkai/argo-workflows that referenced this issue Sep 2, 2024
jingkkkkai added a commit to jingkkkkai/argo-workflows that referenced this issue Sep 2, 2024
@agilgur5 agilgur5 changed the title env http_proxy in wait/init container does not work for oss client env http_proxy in wait/init container does not work for oss client Sep 10, 2024
Joibel pushed a commit to pipekit/argo-workflows that referenced this issue Sep 19, 2024
Joibel pushed a commit that referenced this issue Sep 20, 2024
@agilgur5 agilgur5 added this to the v3.5.x patches milestone Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important type/bug
Projects
None yet
3 participants