Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bigquery max_staleness configuration #231

Closed
sebastiancharrier opened this issue Aug 25, 2023 · 3 comments · Fixed by #237
Closed

Bigquery max_staleness configuration #231

sebastiancharrier opened this issue Aug 25, 2023 · 3 comments · Fixed by #237
Labels
bug Something isn't working Stale triage
Milestone

Comments

@sebastiancharrier
Copy link

Describe the bug

Bigquery specific.
One of the options for external tables is the metadata cache mode. In order to use this option is necessary to add the max_staleness interval.
Intervals in Bigquery are passed in the format of INTERVAL 4 HOUR.
All the options detected as string are passed with single quotes to bigquery 'INTERVAL 4 HOUR'
Unfortunately this specific field return an error if is passed with quotes

Steps to reproduce

- name: my_table external: connection: 'aws-us-east-1.s3-data-lake' location: 's3://data/*' options: format: parquet metadata_cache_mode: automatic max_staleness: INTERVAL 4 HOUR

this is executed in bigquery

create or replace external tablemy_project.my_dataset.my_tablewith connection aws-us-east-1.s3-data-lake options ( uris = ['s3://data/*'], format = 'parquet', metadata_cache_mode = 'automatic', max_staleness = ''INTERVAL 6 HOUR'')

Expected results

max_staleness needs to be passed without quotes

create or replace external tablemy_project.my_dataset.my_tablewith connection aws-us-east-1.s3-data-lake options ( uris = ['s3://data/*'], format = 'parquet', metadata_cache_mode = 'automatic', max_staleness = INTERVAL 6 HOUR)

Actual results

create or replace external tablemy_project.my_dataset.my_tablewith connection aws-us-east-1.s3-data-lake options ( uris = ['s3://data/*'], format = 'parquet', metadata_cache_mode = 'automatic', max_staleness = ''INTERVAL 6 HOUR'')

Error return by Bigquery:

  • Option max_staleness value has type STRING which cannot be coerced to expected type INTERVAL at [9:158]

System information

`packages:

  • package: dbt-labs/dbt_external_tables
    version: 0.8.6`

Which database are you using dbt with?
Bigquery

The output of dbt --version:

1.5.0
Copy link

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

@github-actions github-actions bot added the Stale label Feb 21, 2024
Copy link

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 28, 2024
@dataders dataders reopened this Apr 5, 2024
@dataders
Copy link
Collaborator

dataders commented Apr 5, 2024

resolved by: #237

@dataders dataders closed this as completed Apr 5, 2024
@dataders dataders linked a pull request Apr 5, 2024 that will close this issue
3 tasks
@dataders dataders added this to the 0.9.0 milestone Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Stale triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants