/api/v1/workflows/{namespace}/{name}/log gives an empty response

[Iqqdd99]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

The endpoint /api/v1/workflows/{namespace}/{name}/log doesn't work, it returns an empty response.

The argo CLI works:

$ argo -n argo logs argosay-ghqff
argosay-ghqff:  _____________
argosay-ghqff: < hello world >
argosay-ghqff:  -------------
argosay-ghqff:         \   ^__^
argosay-ghqff:          \  (oo)\_______
argosay-ghqff:             (__)\       )\/\
argosay-ghqff:                 ||----w |
argosay-ghqff:                 ||     ||
argosay-ghqff: time="2024-07-23T08:27:39.514Z" level=info msg="sub-process exited" argo=true error="<nil>"

The data of the workflow can be accessed using the API REST /api/v1/workflows/{namespace}/{name}:

curl -X 'GET' \
  'http://localhost:2746/api/v1/workflows/argo/argosay-ghqff' \
  -H 'accept: application/json' \
  -H "Authorization: $ARGO_TOKEN"
{"metadata":{"name":"argosay-ghqff","generateName":"argosay-","namespace":"argo","uid":"07045089-a54f-47fa-9c76-c35c1f1fbc61","resourceVersion":"31393831","generation":4,"creationTimestamp":"2024-07-23T08:27:24Z","labels":{"submit-from-ui":"true","workflows.argoproj.io/completed":"true","workflows.argoproj.io/creator":"system-serviceaccount-argo-job-creator","workflows.argoproj.io/phase":"Succeeded","workflows.argoproj.io/workflow-template":"argosay"},"annotations":{"workflows.argoproj.io/pod-name-format":"v2"},"managedFields":[{"manager":"argo","operation":"Update","apiVersion":"argoproj.io/v1alpha1","time":"2024-07-23T08:27:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:submit-from-ui":{},"f:workflows.argoproj.io/creator":{},"f:workflows.argoproj.io/workflow-template":{}}},"f:spec":{}}},{"manager":"workflow-controller","operation":"Update","apiVersion":"argoproj.io/v1alpha1","time":"2024-07-23T08:27:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:workflows.argoproj.io/pod-name-format":{}},"f:labels":{"f:workflows.argoproj.io/completed":{},"f:workflows.argoproj.io/phase":{}}},"f:status":{}}}]},"spec":{"arguments":{},"workflowTemplateRef":{"name":"argosay"}},"status":{"phase":"Succeeded","startedAt":"2024-07-23T08:27:24Z","finishedAt":"2024-07-23T08:27:45Z","progress":"1/1","nodes":{"argosay-ghqff":{"id":"argosay-ghqff","name":"argosay-ghqff","displayName":"argosay-ghqff","type":"Pod","templateName":"main","templateScope":"local/","phase":"Succeeded","startedAt":"2024-07-23T08:27:24Z","finishedAt":"2024-07-23T08:27:40Z","progress":"1/1","resourcesDuration":{"cpu":0,"memory":3},"outputs":{"exitCode":"0"},"hostNodeName":"oneke-ip-10-195-45-17"}},"storedTemplates":{"namespaced/argosay/main":{"name":"main","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"argoproj/argosay:v2","command":["cowsay"],"args":["hello world"],"resources":{}}}},"conditions":[{"type":"PodRunning","status":"False"},{"type":"Completed","status":"True"}],"resourcesDuration":{"cpu":0,"memory":3},"storedWorkflowTemplateSpec":{"templates":[{"name":"main","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"argoproj/argosay:v2","command":["cowsay"],"args":["hello world"],"resources":{}}}],"entrypoint":"main","arguments":{},"workflowTemplateRef":{"name":"argosay"}},"artifactRepositoryRef":{"default":true,"artifactRepository":{}},"artifactGCStatus":{"notSpecified":true},"taskResultsCompletionStatus":{"argosay-ghqff":true}}}%

But /api/v1/workflows/{namespace}/{name}/log gives an empty response

$ curl -v -X 'GET' \
  'http://localhost:2746/api/v1/workflows/argo/argosay-ghqff/log' \
  -H 'accept: application/json' \
  -H "Authorization: $ARGO_TOKEN"
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying [::1]:2746...
* Connected to localhost (::1) port 2746
> GET /api/v1/workflows/argo/argosay-ghqff/log HTTP/1.1
> Host: localhost:2746
> User-Agent: curl/8.4.0
> accept: application/json
> Authorization: {Bearer ...}
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Grpc-Metadata-Content-Type: application/grpc
< X-Ratelimit-Limit: 1000
< X-Ratelimit-Remaining: 999
< X-Ratelimit-Reset: Tue, 23 Jul 2024 10:08:41 UTC
< Date: Tue, 23 Jul 2024 10:08:41 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact

Version(s)

sha256:651e199a20c96319c4bd9d24c084a1de9f2e62725749a9f239adfb785147299b

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

metadata:
  name: argo-say
  namespace: argo
  labels:
    example: 'true'
    workflows.argoproj.io/creator: system-serviceaccount-argo-job-creator
spec:
  templates:
    - name: argosay
      inputs:
        parameters:
          - name: message
            value: '{{workflow.parameters.message}}'
      container:
        name: main
        image: argoproj/argosay:v2
        command:
          - /argosay
        args:
          - echo
          - '{{inputs.parameters.message}}'
  entrypoint: argosay
  arguments:
    parameters:
      - name: message
        value: hello argo
  ttlStrategy:
    secondsAfterCompletion: 300
  podGC:
    strategy: OnPodCompletion
  workflowMetadata:
    labels:
      example: 'true'

Logs from the workflow controller

The workflow runs without problems.

Logs from in your workflow's wait container

The workflow runs without problems.

[agilgur5]

651e199a20c96319c4bd9d24c084a1de9f2e62725749a9f239adfb785147299b

That commit SHA 404s for this repo: 651e199

[agilgur5]

Oh, this is an image digest: https://hub.docker.com/layers/argoproj/workflow-controller/latest/images/sha256-651e199a20c96319c4bd9d24c084a1de9f2e62725749a9f239adfb785147299b

[chengjoey]

< Grpc-Metadata-Content-Type: application/grpc

argo-workflows/cmd/argo/commands/common/logs.go

Lines 14 to 35 in ddbb3c7

	func LogWorkflow(ctx context.Context, serviceClient workflowpkg.WorkflowServiceClient, namespace, workflow, podName, grep, selector string, logOptions *corev1.PodLogOptions) {
	// logs
	stream, err := serviceClient.WorkflowLogs(ctx, &workflowpkg.WorkflowLogRequest{
	Name: workflow,
	Namespace: namespace,
	PodName: podName,
	LogOptions: logOptions,
	Selector: selector,
	Grep: grep,
	})
	errors.CheckError(err)
	// loop on log lines
	for {
	event, err := stream.Recv()
	if err == io.EOF {
	return
	}
	errors.CheckError(err)
	fmt.Println(ansiFormat(fmt.Sprintf("%s: %s", event.PodName, event.Content), ansiColorCode(event.PodName)))
	}
	}

argo use grpc client, this is the interface of grpc protocol

argo-server seems compatible with both http and grpc, and /api/v1/workflows/{namespace} seems to work fine

[agilgur5]

I believe this requires ?logOptions.container=main or another container selection

[tooptoop4]

this dupe of #13585