Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yaml selector in manifest json and dbt docs visualization #533

Open
mithungame opened this issue Oct 25, 2024 · 1 comment
Open

yaml selector in manifest json and dbt docs visualization #533

mithungame opened this issue Oct 25, 2024 · 1 comment
Labels
enhancement New feature or request triage

Comments

@mithungame
Copy link

Describe the feature

As the number of models grow increasingly we are relying more and more on yaml selectors instead of the command line select . we are able to list is but it is not so easy to visualize these selectors from dbt docs . Adding a visual dag for yaml selector will be very useful for all growing dbt projects in terms of number of models

Describe alternatives you've considered

dbt list

Additional context

the search ended up taking to different links , but as of now this is what we observe currently
#119
dbt-labs/dbt-core#2693
dbt-labs/dbt-core#3873

yaml selector

  - name: foo
    definition:
      method: path
      value: models/staging

list in command line works

12:43:57  Running with dbt=1.8.3
12:43:58  Registered adapter: snowflake=1.8.3
...
dbt_reference_implementation.staging.test_model

but cannot visualize in dag
also selector is empty in manifest json

    "selectors": {},
    "disabled": {`

Who will this benefit?

All dbt projects with growing number of models

@mithungame mithungame added enhancement New feature or request triage labels Oct 25, 2024
@Yudizz
Copy link

Yudizz commented Nov 7, 2024

Adding reference to related discussion
https://github.com/dbt-labs/dbt-core/discussions/5096

I was able to see selectors listed in manifest.json.

It's listed as an option in dbt docs generate --help but the doc site shows all the nodes instead of the selected ones

Usage: dbt docs generate [OPTIONS]

  Generate the documentation website for your project

Options:
  --cache-selected-only / --no-cache-selected-only
                                  At start of run, populate relational cache
                                  only for schemas containing selected nodes,
                                  or for all schemas of interest.
  -d, --debug / --no-debug        Display debug logging during dbt execution.
                                  Useful for debugging and making bug reports.
  --defer / --no-defer            If set, resolve unselected nodes by
                                  deferring to the manifest within the --state
                                  directory.
  --defer-state DIRECTORY         Override the state directory for deferral
                                  only.
  --deprecated-favor-state TEXT   Internal flag for deprecating old env var.
  -x, --fail-fast / --no-fail-fast
                                  Stop execution on first failure.
  --favor-state / --no-favor-state
                                  If set, defer to the argument provided to
                                  the state flag for resolving unselected
                                  nodes, even if the node(s) exist as a
                                  database object in the current environment.
  --indirect-selection [eager|cautious|buildable|empty]
                                  Choose which tests to select that are
                                  adjacent to selected resources. Eager is
                                  most inclusive, cautious is most exclusive,
                                  and buildable is in between. Empty includes
                                  no tests at all.
  --log-cache-events / --no-log-cache-events
                                  Enable verbose logging for relational cache
                                  events to help when debugging.
  --log-format [text|debug|json|default]
                                  Specify the format of logging to the console
                                  and the log file. Use --log-format-file to
                                  configure the format for the log file
                                  differently than the console.
  --log-format-file [text|debug|json|default]
                                  Specify the format of logging to the log
                                  file by overriding the default value and the
                                  general --log-format setting.
  --log-level [debug|info|warn|error|none]
                                  Specify the minimum severity of events that
                                  are logged to the console and the log file.
                                  Use --log-level-file to configure the
                                  severity for the log file differently than
                                  the console.
  --log-level-file [debug|info|warn|error|none]
                                  Specify the minimum severity of events that
                                  are logged to the log file by overriding the
                                  default value and the general --log-level
                                  setting.
  --log-path PATH                 Configure the 'log-path'. Only applies this
                                  setting for the current run. Overrides the
                                  'DBT_LOG_PATH' if it is set.
  --partial-parse / --no-partial-parse
                                  Allow for partial parsing by looking for and
                                  writing to a pickle file in the target
                                  directory. This overrides the user
                                  configuration file.
  --populate-cache / --no-populate-cache
                                  At start of run, use `show` or
                                  `information_schema` queries to populate a
                                  relational cache, which can speed up
                                  subsequent materializations.
  --print / --no-print            Output all {{ print() }} macro calls.
  --printer-width INTEGER         Sets the width of terminal output
  --profile TEXT                  Which existing profile to load. Overrides
                                  setting in dbt_project.yml.
  -q, --quiet / --no-quiet        Suppress all non-error logging to stdout.
                                  Does not affect {{ print() }} macro calls.
  -r, --record-timing-info PATH   When this option is passed, dbt will output
                                  low-level timing stats to the specified
                                  file. Example: `--record-timing-info
                                  output.profile`
  --send-anonymous-usage-stats / --no-send-anonymous-usage-stats
                                  Send anonymous usage stats to dbt Labs.
  --state DIRECTORY               Unless overridden, use this state directory
                                  for both state comparison and deferral.
  --static-parser / --no-static-parser
                                  Use the static parser.
  -t, --target TEXT               Which target to load for the given profile
  --use-colors / --no-use-colors  Specify whether log output is colorized in
                                  the console and the log file. Use --use-
                                  colors-file/--no-use-colors-file to colorize
                                  the log file differently than the console.
  --use-colors-file / --no-use-colors-file
                                  Specify whether log file output is colorized
                                  by overriding the default value and the
                                  general --use-colors/--no-use-colors
                                  setting.
  --use-experimental-parser / --no-use-experimental-parser
                                  Enable experimental parsing features.
  -V, -v, --version               Show version information and exit
  --version-check / --no-version-check
                                  If set, ensure the installed dbt version
                                  matches the require-dbt-version specified in
                                  the dbt_project.yml file (if any).
                                  Otherwise, allow them to differ.
  --warn-error                    If dbt would normally warn, instead raise an
                                  exception. Examples include --select that
                                  selects nothing, deprecations,
                                  configurations with no associated models,
                                  invalid test configurations, and missing
                                  sources/refs in tests.
  --warn-error-options WARNERROROPTIONSTYPE
                                  If dbt would normally warn, instead raise an
                                  exception based on include/exclude
                                  configuration. Examples include --select
                                  that selects nothing, deprecations,
                                  configurations with no associated models,
                                  invalid test configurations, and missing
                                  sources/refs in tests. This argument should
                                  be a YAML string, with keys 'include' or
                                  'exclude'. eg. '{"include": "all",
                                  "exclude": ["NoNodesForSelectionCriteria"]}'
  --write-json / --no-write-json  Whether or not to write the manifest.json
                                  and run_results.json files to the target
                                  directory
  --compile / --no-compile        Whether or not to run 'dbt compile' as part
                                  of docs generation
  --exclude TUPLE                 Specify the nodes to exclude.
  --profiles-dir PATH             Which directory to look in for the
                                  profiles.yml file. If not set, dbt will look
                                  in the current working directory first, then
                                  HOME/.dbt/
  --project-dir PATH              Which directory to look in for the
                                  dbt_project.yml file. Default is the current
                                  working directory and its parents.
  -s, -m, --select, --models, --model TUPLE
                                  Specify the nodes to include.
  --selector TEXT                 The selector name to use, as defined in
                                  selectors.yml
  --empty-catalog                 If specified, generate empty catalog.json
                                  file during the `dbt docs generate` command.
  --static                        Generate an additional static_index.html
                                  with manifest and catalog built-in.
  --target-path PATH              Configure the 'target-path'. Only applies
                                  this setting for the current run. Overrides
                                  the 'DBT_TARGET_PATH' if it is set.
  --threads INTEGER               Specify number of threads to use while
                                  executing models. Overrides settings in
                                  profiles.yml.
  --vars YAML                     Supply variables to the project. This
                                  argument overrides variables defined in your
                                  dbt_project.yml file. This argument should
                                  be a YAML string, eg. '{my_variable:
                                  my_value}'
  -h, --help                      Show this message and exit. 

Would love to see this is implemented or would be even better if a selector tab is added to the doc site Lineage Graph page

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request triage
Projects
None yet
Development

No branches or pull requests

2 participants