GCS Copy unable to copy from local storage files

[shrutimantri]

Expected Behavior

Given a file to be copied from local storage to GCS, the file should get copied to the appropriate GCS bucket destination successfully.
Note: The Copy task works well for copying files from one GCS location into another, but is unable to copy from local storage into GCS bucket.

Actual Behaviour

As per the description of this task on this page, the file present in Kestra's internal storage can be copied to GCS location.

But I get the following error while executing the flow with the Copy task:

2024-01-29 15:23:23.834 • Using service account: kestra-gcs-svc-account@shruti-demo-project.iam.gserviceaccount.com
2024-01-29 15:23:23.837Moving from 'kestra:///company/team/gcp-gcs-copy/executions/63mIJjgxO0uzupqdl5mRdr/tasks/http-download/4ZediJHm6aN6YgMZCNx4Fm/7040886805462740981.csv' to 'gs://smantri-kestra-bucket/test/salaries/salaries.csv'
2024-01-29 15:23:25.178The billing account for the owning project is disabled in state closed
2024-01-29 15:23:25.178403 Forbidden
POST https://storage.googleapis.com/storage/v1/b/kestra/o/company%2Fteam%2Fgcp-gcs-copy%2Fexecutions%2F63mIJjgxO0uzupqdl5mRdr%2Ftasks%2Fhttp-download%2F4ZediJHm6aN6YgMZCNx4Fm%2F7040886805462740981.csv/rewriteTo/b/smantri-kestra-bucket/o/test%2Fsalaries%2Fsalaries.csv?projection=full
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "location" : "Authorization",
    "locationType" : "header",
    "message" : "The billing account for the owning project is disabled in state closed",
    "reason" : "accountDisabled"
  } ],
  "message" : "The billing account for the owning project is disabled in state closed"
}
2024-01-29 15:23:25.178com.google.cloud.storage.StorageException: The billing account for the owning project is disabled in state closed
	at com.google.cloud.storage.StorageException.translate(StorageException.java:165)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:313)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.rewrite(HttpStorageRpc.java:1162)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.openRewrite(HttpStorageRpc.java:1090)
	at com.google.cloud.storage.StorageImpl.lambda$copy$18(StorageImpl.java:593)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.Retrying.run(Retrying.java:65)
	at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1515)
	at com.google.cloud.storage.StorageImpl.copy(StorageImpl.java:591)
	at io.kestra.plugin.gcp.gcs.Copy.run(Copy.java:77)
	at io.kestra.plugin.gcp.gcs.Copy.run(Copy.java:21)
	at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:684)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
POST https://storage.googleapis.com/storage/v1/b/kestra/o/company%2Fteam%2Fgcp-gcs-copy%2Fexecutions%2F63mIJjgxO0uzupqdl5mRdr%2Ftasks%2Fhttp-download%2F4ZediJHm6aN6YgMZCNx4Fm%2F7040886805462740981.csv/rewriteTo/b/smantri-kestra-bucket/o/test%2Fsalaries%2Fsalaries.csv?projection=full
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "location" : "Authorization",
    "locationType" : "header",
    "message" : "The billing account for the owning project is disabled in state closed",
    "reason" : "accountDisabled"
  } ],
  "message" : "The billing account for the owning project is disabled in state closed"
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.rewrite(HttpStorageRpc.java:1152)
	... 11 more

Steps To Reproduce

1. Create a flow.
2. First task in the flow can be to download a file from http. This downloaded file can be used to copy to GCS.
3. Second task will be the Copy task with from being the internal storage file that has got downloaded from the first task. You can copy this file to any GCS location.
4. Execute the flow.

Environment Information

• Kestra Version: 0.13.8
• Plugin version: 0.13.8
• Operating System (OS / Docker / Kubernetes): Docker
• Java Version (If not docker):

Example flow

id: gcp-gcs-copy
namespace: company.team
tasks:
  - id: http_download
    type: io.kestra.plugin.fs.http.Download
    uri: https://raw.githubusercontent.com/kestra-io/datasets/main/csv/salaries.csv
  - id: "copy"
    type: "io.kestra.plugin.gcp.gcs.Copy"
    serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
    projectId: "<project_namw>"
    from: "{{ outputs.http_download.uri }}"
    to: "gs://<bucket_name>/test/salaries/salaries.csv"

[loicmathieu]

I think the description is wrong, if you want to send an internal storage file to a GCS bucket you need to use the Upload task. This task is only to copy files between GCS buckets.