Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PMM-13171 Add ClickHouse credentials #3646

Open
wants to merge 22 commits into
base: v3
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
c8f4e00
PMM-13171 Update Readme
ademidoff Feb 13, 2025
9878b9e
PMM-13171 Add credentials via env variables
ademidoff Feb 13, 2025
e03e35c
PMM-13171 Use v1 in protobuf names
ademidoff Feb 13, 2025
cd382b4
PMM-13171 Add empty line
ademidoff Feb 13, 2025
f10146f
PMM-13171 Remove redundant const
ademidoff Feb 13, 2025
d21f9bf
PMM-13171 Pass CH creds to qan-api spec
ademidoff Feb 13, 2025
d7eb535
PMM-13171 Align parameters
ademidoff Feb 13, 2025
26ebc09
Merge branch 'v3' into PMM-13171-add-clickhouse-credentials
ademidoff Feb 13, 2025
4ada2cb
PMM-13171 Fix the spelling
ademidoff Feb 13, 2025
448eaa0
PMM-13171 Fix the test
ademidoff Feb 14, 2025
790399d
PMM-13171 Add user password and change users' config
ademidoff Feb 14, 2025
13b12d1
PMM-13171 Fix the tests
ademidoff Feb 14, 2025
6d12e8b
PMM-13171 Fix the typo
ademidoff Feb 14, 2025
604f7b1
PMM-13171 Remove logging clickhouse password
ademidoff Feb 14, 2025
6cc7dde
PMM-13171 Add creds to clickhouse telemetry
ademidoff Feb 14, 2025
b7f9fc0
PMM-13171 Do not log clickhouse password
ademidoff Feb 15, 2025
683507e
PMM-13171 Update docker-compose.yml
ademidoff Feb 15, 2025
62f1fd5
PMM-13171 Update the documentation
ademidoff Feb 15, 2025
155a900
PMM-13171 Update the HA documentation
ademidoff Feb 15, 2025
9c7ef1b
Merge branch 'v3' into PMM-13171-add-clickhouse-credentials
ademidoff Feb 18, 2025
6786624
Merge branch 'v3' into PMM-13171-add-clickhouse-credentials
ademidoff Feb 20, 2025
274f67e
Merge branch 'v3' into PMM-13171-add-clickhouse-credentials
ademidoff Mar 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/managed.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,10 @@ jobs:
run: docker image rm ${{ env.PMM_CACHE_IMAGE }}

- name: Rebuild the binaries
run: docker exec -i pmm-server make run-managed-ci run-agent run-vmproxy run-qan
run: |
# TODO: remove the line below before merging
docker cp ./build/ansible/roles/clickhouse/files/users.xml pmm-server:/etc/clickhouse-server/users.xml
docker exec -i pmm-server make run-managed-ci run-agent run-vmproxy run-qan

- name: Check the status of components
continue-on-error: true
Expand Down
8 changes: 4 additions & 4 deletions build/ansible/roles/clickhouse/files/config.xml
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@
<!-- Compatibility with MySQL protocol.
ClickHouse will pretend to be MySQL for applications connecting to this port.
-->
<mysql_port>9004</mysql_port>
<!-- <mysql_port>9004</mysql_port> -->
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We likely don't need to have that many ports open. Please review this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't use it that way I think - so fair point.


<!-- Compatibility with PostgreSQL protocol.
ClickHouse will pretend to be PostgreSQL for applications connecting to this port.
Expand Down Expand Up @@ -484,9 +484,9 @@
<tmp_path>/srv/clickhouse/tmp/</tmp_path>

<!-- Disable AuthType plaintext_password and no_password for ACL. -->
<allow_plaintext_password>1</allow_plaintext_password>
<allow_no_password>1</allow_no_password>
<allow_implicit_no_password>1</allow_implicit_no_password>
<allow_plaintext_password>0</allow_plaintext_password>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will harden security, for it was too poor before.

<allow_no_password>0</allow_no_password>
<allow_implicit_no_password>0</allow_implicit_no_password>

<!-- When a user does not specify a password type in the CREATE USER query, the default password type is used.
Accepted values are: 'plaintext_password', 'sha256_password', 'double_sha1_password', 'bcrypt_password'.
Expand Down
110 changes: 110 additions & 0 deletions build/ansible/roles/clickhouse/files/users.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
<clickhouse>
<!-- See also the files in users.d directory where the settings can be overridden. -->

<!-- Profiles of settings. -->
<profiles>
<!-- Default settings. -->
<default>
</default>

<!-- Profile that allows only read queries. -->
<readonly>
<readonly>1</readonly>
</readonly>
</profiles>

<!-- Users and ACL. -->
<users>
<!-- If user name was not specified, 'default' user is used. -->
<default>
<!-- See also the files in users.d directory where the password can be overridden.

Password could be specified in plaintext or in SHA256 (in hex format).

If you want to specify password in plaintext (not recommended), place it in 'password' element.
Example: <password>qwerty</password>.
Password could be empty.

If you want to specify SHA256, place it in 'password_sha256_hex' element.
Example: <password_sha256_hex>65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5</password_sha256_hex>
Restrictions of SHA256: impossibility to connect to ClickHouse using MySQL JS client (as of July 2019).

If you want to specify double SHA1, place it in 'password_double_sha1_hex' element.
Example: <password_double_sha1_hex>e395796d6546b1b65db9d665cd43f0e858dd4303</password_double_sha1_hex>

If you want to specify a previously defined LDAP server (see 'ldap_servers' in the main config) for authentication,
place its name in 'server' element inside 'ldap' element.
Example: <ldap><server>my_ldap_server</server></ldap>

If you want to authenticate the user via Kerberos (assuming Kerberos is enabled, see 'kerberos' in the main config),
place 'kerberos' element instead of 'password' (and similar) elements.
The name part of the canonical principal name of the initiator must match the user name for authentication to succeed.
You can also place 'realm' element inside 'kerberos' element to further restrict authentication to only those requests
whose initiator's realm matches it.
Example: <kerberos />
Example: <kerberos><realm>EXAMPLE.COM</realm></kerberos>

How to generate decent password:
Execute: PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'
In first line will be password and in second - corresponding SHA256.

How to generate double SHA1:
Execute: PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha1sum | tr -d '-' | xxd -r -p | sha1sum | tr -d '-'
In first line will be password and in second - corresponding double SHA1.
-->
<password_sha256_hex>7e099f39b84ea79559b3e85ea046804e63725fd1f46b37f281276aae20f86dc3</password_sha256_hex>

<!-- List of networks with open access.

To open access from everywhere, specify:
<ip>::/0</ip>

To open access only from localhost, specify:
<ip>::1</ip>
<ip>127.0.0.1</ip>

Each element of list has one of the following forms:
<ip> IP-address or network mask. Examples: 213.180.204.3 or 10.0.0.1/8 or 10.0.0.1/255.255.255.0
2a02:6b8::3 or 2a02:6b8::3/64 or 2a02:6b8::3/ffff:ffff:ffff:ffff::.
<host> Hostname. Example: server01.clickhouse.com.
To check access, DNS query is performed, and all received addresses compared to peer address.
<host_regexp> Regular expression for host names. Example, ^server\d\d-\d\d-\d\.clickhouse\.com$
To check access, DNS PTR query is performed for peer address and then regexp is applied.
Then, for result of PTR query, another DNS query is performed and all received addresses compared to peer address.
Strongly recommended that regexp is ends with $
All results of DNS requests are cached till server restart.
-->
<networks>
<ip>::/0</ip>
</networks>

<!-- Settings profile for user. -->
<profile>default</profile>

<!-- Quota for user. -->
<quota>default</quota>

<!-- User can create other users and grant rights to them. -->
<!-- <access_management>1</access_management> -->
</default>
</users>

<!-- Quotas. -->
<quotas>
<!-- Name of quota. -->
<default>
<!-- Limits for time interval. You could specify many intervals with different limits. -->
<interval>
<!-- Length of interval. -->
<duration>3600</duration>

<!-- No limits. Just calculate resource usage for time interval. -->
<queries>0</queries>
<errors>0</errors>
<result_rows>0</result_rows>
<read_rows>0</read_rows>
<execution_time>0</execution_time>
</interval>
</default>
</quotas>
</clickhouse>
9 changes: 6 additions & 3 deletions build/ansible/roles/clickhouse/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,16 @@
- /var/log/clickhouse-server
- /run/clickhouse-server

- name: Copy a customized clickhouse config
- name: Copy customized clickhouse config files
copy:
src: config.xml
dest: /etc/clickhouse-server/config.xml
src: "{{ item }}"
dest: "/etc/clickhouse-server/{{ item }}"
owner: pmm
group: pmm
mode: 0664
loop:
- config.xml
- users.xml

# We need to remove capabilities because we run PMM in an unprivileged container
# But we run clickhouse as root user
Expand Down
12 changes: 10 additions & 2 deletions build/ansible/roles/grafana/files/datasources.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ apiVersion: 1
deleteDatasources:
- name: ClickHouse
orgId: 1

datasources:
- name: Metrics
version: 2
Expand All @@ -14,6 +15,7 @@ datasources:
httpMethod: POST
keepCookies: []
timeInterval: 1s

- name: PostgreSQL
version: 2
orgId: 1
Expand All @@ -30,16 +32,22 @@ datasources:
sslmode: ${PMM_POSTGRES_SSL_MODE}
secureJsonData:
password: ${PMM_POSTGRES_DBPASSWORD}

- name: PTSummary
version: 2
orgId: 1
type: pmm-pt-summary-datasource
access: proxy

# https://github.com/grafana/clickhouse-datasource?tab=readme-ov-file#with-a-configuration-file
- name: ClickHouse
orgId: 1
version: 2
type: grafana-clickhouse-datasource
jsonData:
username: ${PMM_CLICKHOUSE_USERNAME}
port: ${PMM_CLICKHOUSE_PORT}
server: ${PMM_CLICKHOUSE_HOST}
tlsSkipVerify: false
host: ${PMM_CLICKHOUSE_HOST}
tlsSkipVerify: false
secureJsonData:
password: ${PMM_CLICKHOUSE_PASSWORD}
4 changes: 3 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,14 @@ services:
# - PMM_DEV_TELEMETRY_RETRY_BACKOFF=10s
# - PMM_CLICKHOUSE_ADDR=127.0.0.1:9000
# - PMM_CLICKHOUSE_DATABASE=pmm
# - PMM_CLICKHOUSE_USER=default
# - PMM_CLICKHOUSE_PASSWORD=<password>
# - PMM_DEBUG=1
# - PMM_DEV_ADVISOR_CHECKS_FILE=/srv/checks/local-checks.yml
# - PMM_POSTGRES_ADDR=pg
# - PMM_POSTGRES_DBNAME=pmm-managed
# - PMM_POSTGRES_USERNAME=pmm-managed
# - PMM_POSTGRES_DBPASSWORD=pmm-managed
# - PMM_POSTGRES_DBPASSWORD=<password>
# - PMM_POSTGRES_SSL_MODE=verify-full
# - PMM_POSTGRES_SSL_CA_PATH=/tmp/certs/root.crt
# - PMM_POSTGRES_SSL_KEY_PATH=/tmp/certs/pmm-managed.key
Expand Down
14 changes: 11 additions & 3 deletions docs/process/v2_to_v3_environment_variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,12 @@ Below is a list of affected variables and their new names.
| `PERCONA_TEST_CHECKS_PUBLIC_KEY` | | Removed in PMM v3, use `PMM_DEV_PERCONA_PLATFORM_PUBLIC_KEY` |
| `PERCONA_TEST_NICER_API` | | Removed in PMM v3 |
| `PERCONA_TEST_PMM_CLICKHOUSE_ADDR` | `PMM_CLICKHOUSE_ADDR` | |
| `PERCONA_TEST_PMM_CLICKHOUSE_BLOCK_SIZE` | | Removed in PMM v3, because of new clickhouse version. |
| `PERCONA_TEST_PMM_CLICKHOUSE_DATABASE` | `PMM_CLICKHOUSE_DATABASE` | |
| `PERCONA_TEST_PMM_CLICKHOUSE_DATASOURCE` | `PMM_CLICKHOUSE_DATASOURCE` | |
| `PERCONA_TEST_PMM_CLICKHOUSE_HOST` | `PMM_CLICKHOUSE_HOST` | |
| `PERCONA_TEST_PMM_CLICKHOUSE_POOL_SIZE` | | Removed in PMM v3, because of new clickhouse version. |
| `PERCONA_TEST_PMM_CLICKHOUSE_PORT` | `PMM_CLICKHOUSE_PORT` | |
| `PERCONA_TEST_PMM_CLICKHOUSE_POOL_SIZE` | | Removed in PMM v3, because of new clickhouse version. |
| `PERCONA_TEST_PMM_CLICKHOUSE_BLOCK_SIZE` | | Removed in PMM v3, because of new clickhouse version. |
| `PERCONA_TEST_PMM_DISABLE_BUILTIN_CLICKHOUSE` | `PMM_DISABLE_BUILTIN_CLICKHOUSE` | |
| `PERCONA_TEST_PMM_DISABLE_BUILTIN_POSTGRES` | `PMM_DISABLE_BUILTIN_POSTGRES` | |
| `PERCONA_TEST_INTERFACE_TO_BIND` | `PMM_INTERFACE_TO_BIND` | |
Expand All @@ -53,5 +53,13 @@ Below is a list of affected variables and their new names.
| `PMM_TEST_TELEMETRY_FILE` | `PMM_DEV_TELEMETRY_FILE` | |
| `PERCONA_TEST_TELEMETRY_HOST` | `PMM_DEV_TELEMETRY_HOST` | |
| `PERCONA_TEST_TELEMETRY_INTERVAL` | `PMM_DEV_TELEMETRY_INTERVAL` | |
| `PERCONA_TEST_TELEMETRY_RETRY_BACKOFF` | `PMM_DEV_TELEMETRY_RETRY_BACKOFF` | |
| `PERCONA_TEST_TELEMETRY_RETRY_BACKOFF` | `PMM_DEV_TELEMETRY_RETRY_BACKOFF` | |
| `PERCONA_TEST_VERSION_SERVICE_URL` | | Removed in PMM v3, use `PMM_DEV_PERCONA_PLATFORM_ADDRESS` |


## Changes to environment variables in v3+

| PMM 3 | Comments |
|--------------------------------------------|--------------------------------------------------------------|
| `PMM_CLICKHOUSE_USER` | Added in v3.1.0 |
| `PMM_CLICKHOUSE_PASSWORD` | Added in v3.1.0 |
Loading
Loading