dbtcloud_job: required key not provided

[huy-nguyen26]

Describe the bug

I'm attempting to provision my workflows using Terraform for dbtcloud. However, when I use dbtcloud_job, I encounter the following error:

Error message

│ Error: POST url: https://emea.dbt.com/api/v2/accounts/12/jobs/34567/, status: 400, body: {"status":{"code":400,"is_success":false,"user_message":"The request was invalid. Please double check the provided data and try again.","developer_message":""},"data":{"schedule":{"time":{"interval":"required key not provided"}}}}
│ 
│   with module.dbt_teams["abc"].dbtcloud_job.preprod,
│   on modules/dbt-team/job.tf line 45, in resource "dbtcloud_job" "preprod":
│   45: resource "dbtcloud_job" "preprod" {
│ 
╵
╷
│ Error: POST url: https://emea.dbt.com/api/v2/accounts/89/jobs/101112/, status: 400, body: {"status":{"code":400,"is_success":false,"user_message":"The request was invalid. Please double check the provided data and try again.","developer_message":""},"data":{"schedule":{"time":{"interval":"required key not provided"}}}}
│ 
│   with module.dbt_teams["xyz"].dbtcloud_job.preprod,
│   on modules/dbt-team/job.tf line 45, in resource "dbtcloud_job" "preprod":
│   45: resource "dbtcloud_job" "preprod" {
│ 

Resource configuration

In my configuration, I do as follow

terraform {
  required_providers {
    dbtcloud = {
      source  = "dbt-labs/dbtcloud"
      version = "~> 0.2"
    }
    ....
  }
}
...
resource "dbtcloud_job" "preprod" {
  environment_id = split(":", dbtcloud_environment.preprod.id)[1]
  execute_steps  = [
    "dbt run"
  ]
  name           = "preprod"
  project_id     = dbtcloud_project.self.id
  triggers       = {
    "github_webhook": false
    "git_provider_webhook": false
    ....
  }
  target_name = "preprod"
  num_threads = 4

  schedule_type = "custom_cron"
  schedule_cron = "0 22 * * *"

  lifecycle {
    ignore_changes = [
      execute_steps,
    ]
  }
}

Config (please complete the following information):

terraform {
  required_providers {
    dbtcloud = {
      source  = "dbt-labs/dbtcloud"
      version = "~> 0.2"
    }
    ....
  }
}

Additional context

I've discovered that others have encountered a similar issue: dbt-labs/docs.getdbt.com#451, but it seems to be related to API calls rather than Terraform usage. To resume: Here's the problem with the create job API is, its taking the "id" field as an input in body which is required field. We need to give value of job id in "id" field but we would not be able to get it in create API so we need to pass null as a value in "id" field for create API.

How can I solve this problem please? What do I miss please? Thank you

[b-per]

Thanks for the report!

I have tried to replicate it but can't.

Can you provide the full config for triggers? (without the "....")
On my side, I had to add "schedule": true to the list of triggers

  triggers       = {
    "github_webhook": false
    "git_provider_webhook": false
	"schedule": true
  }

Can you also provide the output of terraform version to see the version of the provider you have installed?

[huy-nguyen26]

@b-per Thank for your response,

My full triggers configuration is as follow

  triggers       = {
    "github_webhook": false
    "git_provider_webhook": false
    "schedule": var.preprod_job_cron != null # (could be either true or false in my usecases)
    "custom_branch_only": false
  }

And the terraform version is v1.7.4.

[b-per]

Thanks. Sorry, I would need to get the provider version when running terraform version, not the terraform version.

[huy-nguyen26]

Oh sorry @b-per I misunderstood.
This is actually the provider version when running terraform version

❯ terraform version
Terraform v1.7.4
on darwin_arm64
+ provider registry.terraform.io/databricks/databricks v1.39.0
+ provider registry.terraform.io/dbt-labs/dbtcloud v0.2.22
+ provider registry.terraform.io/integrations/github v5.45.0

Your version of Terraform is out of date! The latest version
is 1.7.5. You can update by downloading from https://www.terraform.io/downloads.html

[b-per]

I tried again but can't replicate the issue.

Could you update your module to add the default value for schedule_interval = 1 (which is the default value).

  schedule_type = "custom_cron"
  schedule_cron = "0 22 * * *"
  schedule_interval = 1

I don't know why it is not getting added in your case...

[huy-nguyen26]

@b-per , yes I tried to add schedule_interval = 1 into my module but it still gave me error. And I also don't know why only certain use cases are encountering errors while others seem to be okay @@

[b-per]

It is very odd, yes...

I am wondering if it happened because the jobs have been modified in a certain way in the UI after they have been created via the API

For now, you could try

terraform apply -replace="module.dbt_teams["abc"].dbtcloud_job.preprod"

but it will change your jobs in case someone manually updated the execute_steps, so take note of those steps beforehand.

I will try again to replicate the problem even if I was not able so far.

[b-per]

Hi @huy-nguyen26 !

Did replacing the job fix your issue for now?

[b-per]

Hi again. Actually, if you haven't replaced the jobs yet, just wait a bit if possible.

I did more testing and saw that there were some issues when modifying existing jobs. I think that I have been able to replicate your use case and I am working on a fix for it.

I plan to have a new release in a day or so.

[b-per]

Hi! I am releasing 0.2.23

I think it should fix the issue you had.

[huy-nguyen26]

Hi @b-per , it's great to hear this news! Could you please provide some insight into what happened behind the scenes, if possible?

In any case, thank you very much !!!

[b-per]

The logic is a bit complex when handling changes to jobs because of how different jobs can be set.

In this particular example, the fix was here.

If a job was at some point set to use time intervals and then was changed with the interval not valid anymore the missing line was not adding the required value in the payload.

I added an integration test about this issue, so hopefully it won't come back later.

[huy-nguyen26]

Many thank @b-per !! I will test it.