Support other JSON schemas for dbt

[yu-iskw]

Solves #3947

Reference

• https://docs.getdbt.com/docs/build/projects
• https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview

[yu-iskw]

I want to make it work even in a nested directory, because users potentially manage a dbt project in a nested directory, not the root directory.

[yu-iskw]

ditto

[yu-iskw]

We can technically configure the directories instead of the default directories respectively. If we want the rules more flexible, we need to parse all YAML files. But, I'm not sure we can apply file match rules like **/*.yml, **/**/*.yml.

[hyperupcall]

Thanks for PR; could the fileMatch be narrowed further to prevent false positives? Ideally, something with "dbt" in its name;

[yu-iskw]

@hyperupcall Sure. I have modified the patterns. Can you take a look again?

cb46470

[yu-iskw]

@hyperupcall Can you give me advice to solve this? Though I have changed the pattern from **/macros/*.yml to macros/*.yml, I got the workflow errorn.

################ Error message
>> fileMatch with directory must start with "**/" => macros/*.yml

[hyperupcall]

@yu-iskw I believe that rule is set up because the glob patterns are recursive by default, so effectively macros/*.yml is the same as **/macros/*.yml. Making it /macros/*.yml should silence the error since the pattern explicitly matches the root. (Same thing with packages.yml, doing /packages.yml makes it unambiguously match the root directory).

But in my opinion, matching every yaml file in snapshots, models, etc. still is way to broad, even if its only at the root directory. I'm somewhat hesitant with packages.yml (something like packages.dbt.yml would be ideal), but I'm okay with matching selectors.yml. If this is merged as-is, I worry there will be too many false positive matches. For example, I myself have written some ad-hoc packages.(json|yaml) files for use in personal projects. @madskristensen what do you think

[joellabes]

Hello! I'm a DX Advocate at dbt Labs (makers of dbt) - the way that the dbt CLI app knows it's in a dbt project is that it can find a dbt_project.yml file at its cwd or parents, unless explicitly specified elsewhere.

Because of this, there isn’t a required folder pattern, but I would guess most project directories would contain dbt in them somewhere (and it'd be a reasonable convention to require for schemastore). Would you be open to a pattern like **/*dbt*/**/models/**/*.{yaml,yml}?

I ran it through Digital Ocean's glob checker and it seems to work:

[joellabes]

Suggested change

	"url": "https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/latest/selectors-latest.json"
	"url": "https://raw.githubusercontent.com/dbt-labs/dbt-jsonschema/main/schemas/latest/dbt_yml_files-latest.json"

[hyperupcall]

find a dbt_project.yml file at its cwd or parents, unless explicitly specified elsewhere

Ideally, schema fileMatches would "activate" and "deactivate" based on the existence of a particular file like dbt_project.yml. I remember this being useful in other cases as well, so I've opened #3965 to track it.

Would you be open to a pattern like **/*dbt*/**/models/**/*.{yaml,yml}?

Yes, that works!

[joellabes]

I can't contribute directly to this branch, but I've opened #3977 to implement the agreed approach

[hyperupcall]

Closing as superseded by #3977