GCS Compose fails with an 404 error

[shrutimantri]

Expected Behavior

Given a GCS directory path containing few files, the files should concatenate into one file, and the concatenated file should be uploaded to the destination.

Actual Behaviour

The flow execution results into an error:

2024-01-29 15:00:07.828 • Using service account: kestra-gcs-svc-account@shruti-demo-project.iam.gserviceaccount.com
2024-01-29 15:00:07.836 • Using service account: kestra-gcs-svc-account@shruti-demo-project.iam.gserviceaccount.com
2024-01-29 15:00:10.163Found '4' blobs from 'gs://smantri-gcs-kestra-test-bucket/test/users/'
2024-01-29 15:00:10.167Compose from '[com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@3221112c, com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@1afa53f0, com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@672893f7, com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@1fb84247]' to 'gs://smantri-kestra-bucket/test/users.csv'
2024-01-29 15:00:10.597Object test/users/ (generation: 0) not found.
2024-01-29 15:00:10.597404 Not Found
POST https://storage.googleapis.com/storage/v1/b/smantri-kestra-bucket/o/test%2Fusers.csv/compose
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Object test/users/ (generation: 0) not found.",
    "reason" : "notFound"
  } ],
  "message" : "Object test/users/ (generation: 0) not found."
}
2024-01-29 15:00:10.597com.google.cloud.storage.StorageException: Object test/users/ (generation: 0) not found.
	at com.google.cloud.storage.StorageException.translate(StorageException.java:165)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:313)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.compose(HttpStorageRpc.java:716)
	at com.google.cloud.storage.StorageImpl.lambda$compose$16(StorageImpl.java:562)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.Retrying.run(Retrying.java:65)
	at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1515)
	at com.google.cloud.storage.StorageImpl.compose(StorageImpl.java:560)
	at io.kestra.plugin.gcp.gcs.Compose.run(Compose.java:102)
	at io.kestra.plugin.gcp.gcs.Compose.run(Compose.java:21)
	at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:684)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://storage.googleapis.com/storage/v1/b/smantri-kestra-bucket/o/test%2Fusers.csv/compose
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Object test/users/ (generation: 0) not found.",
    "reason" : "notFound"
  } ],
  "message" : "Object test/users/ (generation: 0) not found."
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.compose(HttpStorageRpc.java:713)
	... 10 more

Steps To Reproduce

1. Create a bucket, and in a particular folder in that bucket, create a few files that need to be concatenated.
2. Create a new bucket where the concatenated file will be uploaded.
3. Create a flow with the task as shown in the example on this page.
4. Execute the flow.

Environment Information

• Kestra Version:
• Plugin version:
• Operating System (OS / Docker / Kubernetes):
• Java Version (If not docker):

Example flow

id: gcp-gcs-compose
namespace: company.team
tasks:
  - id: "compose"
    type: "io.kestra.plugin.gcp.gcs.Compose"
    serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
    projectId: "<project_name>"
    list: 
      from: "gs://<bucket1>/test/users/"
    to: "gs://<bucket2>/test/users.csv"

This location: gs://<bucket1>/test/users/ contains 3 files: users1.csv, users2.csv, users3.csv