Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added ignore_case for Snowflake #308

Merged
merged 3 commits into from
Oct 25, 2024
Merged

added ignore_case for Snowflake #308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cebfcf4
added ignore_case for Snowflake
cakkinep Jul 18, 2024
093f967
updated for ignore case changes for infer schema
cakkinep Jul 19, 2024
59d3174
Merge branch 'main' into snowflake_ignore_case
dataders Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions integration_tests/models/plugins/snowflake/people_ignore_case.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
SELECT
{{ dbt_utils.star(from=ref('people'), except=['email']) }},
email as Email
FROM {{ ref('people') }}
32 changes: 31 additions & 1 deletion integration_tests/models/plugins/snowflake/snowflake_external.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,4 +200,34 @@ sources:
- first_name
- last_name
- email
- email_domain
- email_domain


- name: people_parquet_infer_schema_ignore_case_unpartitioned
external:
<<: *parquet-people
infer_schema: true
ignore_case: true
tests:
- dbt_utils.equality:
compare_model: ref('people_ignore_case')
compare_columns:
- id
- first_name
- last_name
- email

- name: people_parquet_column_list_ignore_case_partitioned
external:
<<: *parquet-people
partitions: *parts-of-the-people
ignore_case: true
columns: *cols-of-the-people
tests:
- dbt_utils.equality:
compare_model: ref('people_ignore_case')
compare_columns:
- id
- first_name
- last_name
- email
11 changes: 10 additions & 1 deletion macros/plugins/snowflake/create_external_table.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@
{%- set external = source_node.external -%}
{%- set partitions = external.partitions -%}
{%- set infer_schema = external.infer_schema -%}
{%- set ignore_case = external.ignore_case -%}

{% if infer_schema %}
{% set query_infer_schema %}
select * from table( infer_schema( location=>'{{external.location}}', file_format=>'{{external.file_format}}') )
select * from table( infer_schema( location=>'{{external.location}}', file_format=>'{{external.file_format}}', ignore_case=>true) )
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May want to have the ignore_case option here use the ignore_case variable, defaulting to blank or false to match Snowflake defaults.

Something similar to this commit

Copy link
Contributor Author

@cakkinep cakkinep Aug 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not using column name from the results of this ignore_case=>true, instead i am explicitly using GET_IGNORE_CASE function on the inferred columns to prevent any issues , my observation is when using ignore_case=>true snowflake is converting the columns to upper case by default ( snowflake's interpretation of ignoring case) and using them in the value:: reference, this would not get the correct value.

By explicitly using the function , i am able to make it truly ignore case.

corresponding code on line 59 here - https://github.com/dbt-labs/dbt-external-tables/pull/308/files#diff-bab29748b029dbac246005b734e423bdc39eb7735844083cd6902e1afbcab7f7R59

{% endset %}
{% if execute %}
{% set columns_infer = run_query(query_infer_schema) %}
Expand Down Expand Up @@ -40,7 +41,11 @@
{%- if column.expression -%}
{{column.expression}}
{%- else -%}
{%- if ignore_case -%}
{%- set col_id = 'value:c' ~ loop.index if is_csv else 'GET_IGNORE_CASE($1, ' ~ "'"~ column_quoted ~"'"~ ')' -%}
{%- else -%}
{%- set col_id = 'value:c' ~ loop.index if is_csv else 'value:' ~ column_quoted -%}
{%- endif -%}
(case when is_null_value({{col_id}}) or lower({{col_id}}) = 'null' then null else {{col_id}} end)
{%- endif -%}
{%- endset %}
Expand All @@ -50,7 +55,11 @@
{% else %}
{%- for column in columns_infer %}
{%- set col_expression -%}
{%- if ignore_case -%}
{%- set col_id = 'GET_IGNORE_CASE($1, ' ~ "'"~ column[0] ~"'"~ ')' -%}
{%- else -%}
{%- set col_id = 'value:' ~ column[0] -%}
{%- endif -%}
(case when is_null_value({{col_id}}) or lower({{col_id}}) = 'null' then null else {{col_id}} end)
{%- endset %}
{{column[0]}} {{column[1]}} as ({{col_expression}}::{{column[1]}})
Expand Down
Loading