chore(external docs): generate global option configuration automatically from Rust code #22345

titaneric · 2025-02-02T18:35:19Z

Summary

Aiming to solve the issue mentioned in #21348 (comment), and resolve it by generate a global option configuration page by introducing a new global_option configurational component macro. I also update the corresponding Ruby code for generating component docs. It looks worked.

I want to emphasize that this is not a production code but rather that a PoC. I would refactor the code (especially for the Ruby code, first time to write Ruby) until all reviewers would agree my solution.

I guess it would be a very long procedure to review the code and the generating documents, especially for the difference the manually-written configuration page and the generated one from Rust code. I will help them as much as possible.

TODO:

get every reviwers' approvals for the change
introduce global_option configurational component macro for EVERY global option configuration
introduce overlay and decorated page for the configuration page like to one in website/cue/reference/components/transforms/log_to_metric.cue
write dev docs if needed

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

How did you test this PR?

make generate-component-docs
cd website
make run-production-site-locally

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

Please read our Vector contributor resources.
- make check-all is a good command to run locally. This check is
  defined here. Some of these
  checks might not be relevant to your PR. For Rust changes, at the very least you should run:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

Enrichment table

Secret

pront

Thank you @titaneric! This is a reasonable approach and should eliminate a lot of confusion in this area.

pront · 2025-02-03T14:19:59Z

scripts/generate-component-docs.rb

@@ -1728,6 +1728,42 @@ def render_and_import_component_schema(root_schema, schema_name, component_type,
  )
 end

+def render_and_import_global_option_schema(unwrapped_resolved_schema, friendly_name, config_map_path, cue_relative_path)


Tangential: #22350

pront · 2025-02-03T14:21:45Z

src/enrichment_tables/file.rs

@@ -111,6 +112,9 @@ pub struct FileConfig {
    /// [rfc3339]: https://tools.ietf.org/html/rfc3339
    /// [chrono_fmt]: https://docs.rs/chrono/latest/chrono/format/strftime/index.html#specifiers
    #[serde(default)]
+    #[configurable(metadata(
+        docs::additional_props_description = "represent mapped log field names and types."


Nit:

Suggested change

docs::additional_props_description = "represent mapped log field names and types."

docs::additional_props_description = "Represents mapped log field names and types."

We would also benefit from an example here.

pront · 2025-02-03T14:26:05Z

src/enrichment_tables/mod.rs

@@ -18,10 +18,11 @@ pub mod geoip;
 pub mod mmdb;

 /// Configurable enrichment tables.
-#[configurable_component]
+#[configurable_component(global_option("enrichment_tables", "enrichment table type"))]


Just for my understanding, the second string is a description which can be omitted right?

Yes, you are right!
I remove it and re-generate the component docs, and there is no diff at all.

I will remove it.

titaneric · 2025-02-03T15:13:40Z

Hi @pront , I just want to know that should I continue to work for the listed TODO?

I mean updating ALL of the global option with wrap configurable component macro and Ruby code refactor.

I could try incremental change (multiple PRs) to update the support the global configuration to this new form.

pront · 2025-02-03T16:12:49Z

Hi @pront , I just want to know that should I continue to work for the listed TODO?

I mean updating ALL of the global option with wrap configurable component macro

Ah didn't realize we have more places to update. Can you describe a bit more?

and Ruby code refactor.
Yes on the Ruby refactor. (just to be clear the Rust issue is orthogonal you can ignore that)

I could try incremental change (multiple PRs) to update the support the global configuration to this new form.

Up to you 👍

titaneric · 2025-02-03T16:28:23Z

Ah didn't realize we have more places to update. Can you describe a bit more?

I only update the secret and enrichmentable with the new configurable macro, but there are more options such as data_dir, api and so on.

I might introduce the base and overlay global option configuration just like other components do. I will work on it!

pront · 2025-02-03T16:32:39Z

Ah didn't realize we have more places to update. Can you describe a bit more?

I only update the secret and enrichmentable with the new configurable macro, but there are more options such as data_dir, api and so on.

I might introduce the base and overlay global option configuration just like other components do. I will work on it!

Sounds great. Also, cc @jszwedko (just in case you want to take a look)

titaneric · 2025-02-03T17:21:04Z

I might need couple days to work on it. Should I convert to draft and then turn into ready once I finished, or stay current state and comment it in this thread?

pront · 2025-02-03T17:30:39Z

I might need couple days to work on it. Should I convert to draft and then turn into ready once I finished, or stay current state and comment it in this thread?

You can leave it as is. Take as as much time you need and thank you again. Much appreciated.

jszwedko

This may be the first external contribution modifying the docs generation components ❤️ Thanks for diving into this @titaneric

titaneric · 2025-02-05T07:20:53Z

@pront @jszwedko @buraizu , I think I have finished the change. Please review!

Please use difft to review enrichment_table and secret schema change.

git checkout master
./scripts/cue.sh export -e configuration.configuration.enrichment_tables > before-enrichment_tables.json

git checkout feat/global-options-auto-docs
./scripts/cue.sh export -e configuration.configuration.enrichment_tables > after-enrichment_tables.json

difft ./before-enrichment_tables.json ./after-enrichment_tables.json

before-enrichment_tables.json
after-enrichment_tables.json

git checkout master

# note the single form of `secret` instead of `secrets`
./scripts/cue.sh export -e configuration.configuration.secret > before-secrets.json

git checkout feat/global-options-auto-docs
./scripts/cue.sh export -e configuration.configuration.secrets > after-secrets.json

difft ./before-secrets.json ./after-secrets.json

before-secrets.json

after-secrets.json

titaneric · 2025-02-05T07:38:27Z

src/enrichment_tables/file.rs

@@ -76,7 +77,7 @@ pub struct FileConfig {
    /// 1. One of the built-in-formats listed in the `Timestamp Formats` table below.
    /// 2. The [time format specifiers][chrono_fmt] from Rust’s `chrono` library.
    ///
-    /// ### Types
+    /// Types


I removed the header here since it would be displayed on the right side

Interesting, yeah it might better without that header.

pront · 2025-02-05T17:28:56Z

src/secrets/mod.rs

 #[allow(clippy::large_enum_variant)]
-#[configurable_component]
+#[configurable_component(global_option("secrets"))]


One thing that stood out here is that the old docs define secret: https://vector.dev/docs/reference/configuration/global-options/#secret

And the new docs secrets with a s. Although the latter is better, it seems like a breaking change.

Yes, I agree it's indeed a breaking change for documents.

Can we revert it to secret? If I understand correctly, it's just a rename. But if not, we can mark this as a breaking change (fix is trivial)

I would try it, wait a minute

Yes, you are right!
I SHOULD make it consistent to config instead of folder name.

I have renamed it.

pront · 2025-02-05T17:30:31Z

website/cue/reference/configuration.cue

-				}
-			}
-		}
+		// TODO: generate `common` and `required` fields from ruby according to some tags


Are these TODOs something you plan to address in this PR? (it's fine to leave as is, someone can follow up later in another PR)

Yeah, I have some basic thought for how to solve these TODO by introducing a new configurable macro for global options only. I will start to work on it in this weekend.

🙇 No rush, and we can make it a followup PR

titaneric added 6 commits February 2, 2025 17:50

try add new global_options component type

eb2f9a2

Generate enrichment tables global options docs

4d9b69e

Remove plural form for global option component

6da2b73

Re-generate the doc for enrichment table

9514cfd

Generate schema by each field of global option

e189a95

Generate schema for secrets

fe26bc9

github-actions bot added domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation labels Feb 2, 2025

pront approved these changes Feb 3, 2025

View reviewed changes

pront marked this pull request as ready for review February 3, 2025 14:26

pront requested review from a team as code owners February 3, 2025 14:26

titaneric added 2 commits February 3, 2025 23:07

Remove second option for configurable component

53ca580

Fix description for schema defined in file

3dc42be

Remove redundant enrichment table doc

11f09eb

Reference base enrichment table and secret type in global configuration

de3c1e1

buraizu approved these changes Feb 3, 2025

View reviewed changes

jszwedko approved these changes Feb 4, 2025

View reviewed changes

Move common and required fields from base to overlay config

b91c046

pront added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Feb 4, 2025

titaneric added 3 commits February 5, 2025 14:14

Refactor ruby code to generate the global option base schema

81c0198

Fix reference global schema

51d3821

Fix URL link in markdown

e8fe85e

titaneric commented Feb 5, 2025

View reviewed changes

pront reviewed Feb 5, 2025

View reviewed changes

Rename secrets to secret (follow the config)

2efd6fd

pront approved these changes Feb 5, 2025

View reviewed changes

pront enabled auto-merge February 5, 2025 18:05

Fix script check

d777afd

auto-merge was automatically disabled February 6, 2025 01:27
Head branch was pushed to by a user without write access

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(external docs): generate global option configuration automatically from Rust code #22345

chore(external docs): generate global option configuration automatically from Rust code #22345

titaneric commented Feb 2, 2025

pront left a comment

pront Feb 3, 2025

pront Feb 3, 2025 •

edited

Loading

pront Feb 3, 2025

titaneric Feb 3, 2025

titaneric commented Feb 3, 2025 •

edited

Loading

pront commented Feb 3, 2025

titaneric commented Feb 3, 2025

pront commented Feb 3, 2025

titaneric commented Feb 3, 2025

pront commented Feb 3, 2025

jszwedko left a comment

titaneric commented Feb 5, 2025 •

edited

Loading

titaneric Feb 5, 2025

pront Feb 5, 2025

pront Feb 5, 2025

titaneric Feb 5, 2025

pront Feb 5, 2025

titaneric Feb 5, 2025

titaneric Feb 5, 2025

pront Feb 5, 2025

titaneric Feb 5, 2025

pront Feb 5, 2025

	docs::additional_props_description = "represent mapped log field names and types."
	docs::additional_props_description = "Represents mapped log field names and types."

chore(external docs): generate global option configuration automatically from Rust code #22345

Are you sure you want to change the base?

chore(external docs): generate global option configuration automatically from Rust code #22345

Conversation

titaneric commented Feb 2, 2025

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Does this PR include user facing changes?

Checklist

References

pront left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pront Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

titaneric commented Feb 3, 2025 • edited Loading

pront commented Feb 3, 2025

titaneric commented Feb 3, 2025

pront commented Feb 3, 2025

titaneric commented Feb 3, 2025

pront commented Feb 3, 2025

jszwedko left a comment

Choose a reason for hiding this comment

titaneric commented Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pront Feb 3, 2025 •

edited

Loading

titaneric commented Feb 3, 2025 •

edited

Loading

titaneric commented Feb 5, 2025 •

edited

Loading