Auto generate Global Server Settings markdown from source

[Blargian]

Child of #2730. Auto generate Global Server Settings markdown from source.

[gingerwizard]

Not wont cover all settings e.g.

Added in but without docs for the config setting: ClickHouse/ClickHouse#60117

Server setting in config.xml: https://github.com/ClickHouse/ClickHouse/blob/9caac43b2aa5e7c5474a87b7596dea95f5a2569a/programs/server/config.xml#L775-L777

Should be on https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings

[Blargian]

Used to generate markdown file with settings outside of ServerSettings.cpp. We will use it to create a markdown file with combined settings from ServerSettings.cpp and this one.

List of settings outside of ServerSettings.cpp:

access_control_path
asynchronous_insert_log
auth_use_forwarded_address
background_merges_mutations_scheduling_policy
backup_log
builtin_dictionaries_reload_interval
cache_size_to_ram_max_ratio
cgroup_memory_watcher_hard_limit_ratio
cgroup_memory_watcher_soft_limit_ratio
compression
concurrent_threads_soft_limit_ratio_to_cores
core_dump
crash_log
custom_settings_prefixes
database_atomic_delay_before_drop_table_sec
database_catalog_unused_dir_cleanup_period_sec
database_catalog_unused_dir_hide_timeout_sec
database_catalog_unused_dir_rm_timeout_sec
default_database
default_profile
default_replica_name
default_replica_path
dictionaries_config
disable_internal_dns_cache
disable_tunneling_for_https_requests_over_http_proxy
distributed_ddl
dns_cache_update_period
dns_max_consecutive_failures
encryption
error_log
format_schema_path
graphite
graphite_rollup
hsts_max_age
http_server_default_response
include_from
interserver_http_credentials
interserver_http_host
interserver_http_port
interserver_https_host
interserver_https_port
interserver_listen_host
listen_backlog
listen_host
listen_reuse_port
listen_try
logger
macros
max_open_files
max_server_memory_usage_to_ram_ratio
merge_tree
merge_workload
merges_mutations_memory_usage_to_ram_ratio
metric_log
mlock_executable
mutation_workload
mysql_port
openSSL
part_log
path
postgresql_port
proxy
query_cache
query_log
query_masking_rules
query_thread_log
query_views_log
remote_servers
replica_group_name
replicated_merge_tree
send_crash_reports
table_engines_require_grant
tcp_port
tcp_port_secure
temporary_data_in_cache
text_log
timezone
tmp_path
tmp_policy
trace_log
use_minimalistic_part_header_in_zookeeper
user_defined_executable_functions_config
user_defined_path
user_directories
user_files_path
user_scripts_path
users_config
workload_path
workload_zookeeper_path
zookeeper

WITH
    'settings.md' AS doc_file, settings_from_docs AS
    (
    WITH
    arrayJoin(extractAllGroups(raw_blob, '## (\\w+)(?:\\s[^\n]+)?\n\\s+((?:[^#]|#[^#]|##[^ ])+)')) AS g, g[1] AS name, replaceRegexpAll(replaceRegexpAll(g[2], '\n(Type|Default( value)?): [^\n]+\n', ''), '^\n+|\n+$', '') AS doc
    SELECT
    name, doc
    FROM file(doc_file, RawBLOB)
    ORDER BY name ASC
    ),
    settings_outside_source AS
    (
        SELECT line AS name FROM file('settings_outside_server_settings.txt', LineAsString)
    ),
    content AS
    (
        SELECT *
        FROM settings_from_docs
        WHERE name IN settings_outside_source
    ),

    main_content AS
    (
        SELECT format('\n## {}\n\n{}\n', name, doc)
        FROM content
        ORDER BY name
    ),
    '<!-- Note: These settings are used to autogenerate Global Server Settings. They are the server settings which are not found in ServerSettings.cpp -->\n<!-- Please only add settings to this file if they do not exist in core/ServerSettings.cpp -->\n'
    AS prefix
SELECT prefix || (SELECT groupConcat(*) FROM main_content)
INTO OUTFILE 'settings_outside_of_source.md' TRUNCATE FORMAT LineAsString

N.B - don't use trim() function because it messes up the spacing of the markdown tables.

[gingerwizard]

As part of this, we want to generate sublist of beta and experimental settings. This produces a markdown snippets which we can import into a relevant page cc @melvynator @Blargian e.g. beta/experimental page

[Blargian]

As part of this, we want to generate sublist of beta and experimental settings. This produces a markdown snippets which we can import into a relevant page cc @melvynator @Blargian e.g. beta/experimental page

@gingerwizard server settings (both defined in core/ServerSettings.cpp and outside in config.xml) don't currently have a tier level attached to them I believe:

DESCRIBE TABLE system.server_settings

Query id: b3756f6c-bf3b-4a3c-8eee-31ba567c56e9

   ┌─name───────────────────────┬─type───────────────────────────────────────────────────────────────┬─default_type─┬─default_expression─┬─comment─────────────────────────────────────────────┬─codec_expression─┬─ttl_expression─┐
1. │ name                       │ String                                                             │              │                    │ Server setting name.                                │                  │                │
2. │ value                      │ String                                                             │              │                    │ Server setting value.                               │                  │                │
3. │ default                    │ String                                                             │              │                    │ Server setting default value.                       │                  │                │
4. │ changed                    │ UInt8                                                              │              │                    │ Shows whether a setting was specified in config.xml │                  │                │
5. │ description                │ String                                                             │              │                    │ Short server setting description.                   │                  │                │
6. │ type                       │ String                                                             │              │                    │ Server setting value type.                          │                  │                │
7. │ changeable_without_restart │ Enum8('No' = 0, 'IncreaseOnly' = 1, 'DecreaseOnly' = 2, 'Yes' = 3) │              │                    │ Shows whether a setting can be changed at runtime.  │                  │                │
8. │ is_obsolete                │ UInt8                                                              │              │                    │ Shows whether a setting is obsolete.                │                  │                │
   └────────────────────────────┴────────────────────────────────────────────────────────────────────┴──────────────┴────────────────────┴─────────────────────────────────────────────────────┴──────────────────┴────────────────┘

To clarify, is the intention to identify which ones are beta/experimental so that the functionality for tier levels can be added to these? Or is the comment about settings more generally? (not specifically global server settings)

[rschu1ze]

@Blargian A separate settings_outside_server_settings.txt seems too fragile and maintenance-heavy. In #2730 (comment), I suggested to focus on system.server_settings and ignore all other server settings (for the purpose of documentation). The real fix to deal with the rest would be to move them into system.server_settings (i.e. in the C++ code). Sometimes that is simple, sometimes it's not (e.g. for nested settings). In any case, that would be an orthogonal task ...

To clarify, is the intention to identify which ones are beta/experimental so that the functionality for tier levels can be added to these? Or is the comment about settings more generally? (not specifically global server settings)

The experimental/beta badge was recently added for full-blown features here: ClickHouse/ClickHouse#73978. I guess the question is if we should add a corresponding badge to auto-generated setting docs. If you look at src/Core/Settings.cpp, there is a block for experimental features already and each of it is marked EXPERIMENTAL (beta and prod settings are not distinguished). No such block/tag exists in src/Core/ServerSettings.cpp. Afaik, the reason is that there are much fewer server settings and I don't even find an experimental ones.
My recommendation is to ignore experimental badges for server settings. For normal settings, we can generate badges based on field tier in system.settings.

[Blargian]

Hi @rschu1ze Thanks for clarifying. Regardingsettings_outside_server_settings.md above I have already updated system.server_settings with the more complete documentation in #73838. My idea with the list above (still in progress) was to have a hidden markdown file containing the server settings not in the C++ code. We will then auto generate the markdown from the now up to date 'C++ settings' in system.server_settings and the rest which are contained in that hidden file. Having settings_outside_server_settings.md is a temporary solution but I think it will make it easier to see which ones still need to be moved over to 'C++ documentation'. Do you agree?

[rschu1ze]

How is settings_outside_server_settings.md generated? Is it more or less "programs/server/config.xml" minus the settings displayed by system.server_settings? Sorry that I point back to my earlier comment #2730 (comment) again but most settings outside system.server_settings are nested settings. They cannot appear in system.server_settings for principal reasons (at least with the current state of system.server_settings), therefore, it is impossible to auto-generate public docs from them (at least for now).

My proposal is this:

• ditch settings_outside_server_settings,
• instead, try to auto-generate public docs from system.server_settings (i.e. adjust the doc-generating scripts for that),
• this will produce a sub-set of https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings
• the remaining, non-auto-generated settings are already documented as markdown. We can keep these docs as is.

[Blargian]

How is settings_outside_server_settings.md generated? Is it more or less "programs/server/config.xml" minus the settings displayed by system.server_settings? Sorry that I point back to my earlier comment #2730 (comment) again but most settings outside system.server_settings are nested settings. They cannot appear in system.server_settings for principal reasons (at least with the current state of system.server_settings), therefore, it is impossible to auto-generate public docs from them (at least for now).

My proposal is this:

• ditch settings_outside_server_settings,

• instead, try to auto-generate public docs from system.server_settings (i.e. adjust the doc-generating scripts for that),

• this will produce a sub-set of https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings

• the remaining, non-auto-generated settings are already documented as markdown. We can keep these docs as is.

Yes it's all of the settings currently on Global server settings minus those which are in system.server_settings (which are mainly nested settings from config.xml)

It's clear to me. The whole system.server_settingsthing above is more of a note to self so I know which settings to keep and which to replace with autogenerated markdown (via the doc generating script).