Deploy prometheus #1013

HaoYang0000 · 2024-12-24T11:17:00Z

This PR add make target to deploy/undeploy prometheus instance, and its related service monitor, secrets.

roypaulin

Try it with 2 vdb in separate namespaces and let me know if prometheus can scrape their metrics.

Makefile

scripts/deploy-prometheus.sh

Makefile

HaoYang0000 · 2024-12-26T12:24:48Z

Try it with 2 vdb in separate namespaces and let me know if prometheus can scrape their metrics.
Yes I tried to deploy two Vertica db in different namespace, once the two service monitor are deployed in the same namespace as the db, we can get the metrics from both. I will update the confluence page for the steps.

roypaulin

I think we can already add an e2e test, that we will extend later for the autoscaler. For now, you can deploy prometheus, create vdb, create a service monitor and check that prometheus can fetch the metrics from the db: curl http://<prometheus-instance>:9090/api/v1/label/__name__/values and then search for one vertica metric.

Makefile

roypaulin · 2024-12-31T11:22:44Z

helm-charts/prometheus/values.yaml

Given it is not a chart, let's this file from helm-charts/prometheus to prometheus

This file must also be generated by a script so we can parametrize a few of these fields.

The number of replicas must be configurable

We also want to be able to control the resources(cpu, mem) of prometheus pods(server.resources.requests/server.resources.limits)

The file is moved to prometheus.
This file meant to provide a default value for the helm, we can parameterized fields by providing PROMETHEUS_HELM_OVERRIDES=--set attribute.name=something

helm-charts/prometheus/values.yaml

tests/e2e-leg-11/prometheustest/20-wait-for-data-pull.yaml

tests/e2e-leg-11/prometheustest/10-deploy-prometheus.yaml

roypaulin · 2025-01-02T13:55:24Z

tests/e2e-leg-11/prometheustest/15-deploy-prometheus-service-monitor.yaml

Add an assert to check the service monitor

roypaulin · 2025-01-02T13:56:26Z

tests/e2e-leg-11/prometheustest/10-assert.yaml

Check for the deployment instead(all replicas must be available), and also the service.

roypaulin

When uninstalling a prometheus release, we should first clean up all the servicemonitors and secrets tied to that release. You can use kubectl delete servicemonitor/secret -l release=

roypaulin · 2025-01-02T15:58:55Z

Makefile

@@ -623,7 +633,7 @@ uninstall: manifests kustomize ## Uninstall CRDs from the K8s cluster specified
 # If this secret does not exist then it is simply ignored.
 deploy-operator: manifests kustomize ## Using helm or olm, deploy the operator in the K8s cluster
 ifeq ($(DEPLOY_WITH), helm)
-	helm install $(DEPLOY_WAIT) -n $(NAMESPACE) --create-namespace $(HELM_RELEASE_NAME) $(OPERATOR_CHART) --set image.repo=null --set image.name=${OPERATOR_IMG} --set image.pullPolicy=$(HELM_IMAGE_PULL_POLICY) --set imagePullSecrets[0].name=priv-reg-cred --set controllers.scope=$(CONTROLLERS_SCOPE) --set controllers.vdbMaxBackoffDuration=$(VDB_MAX_BACKOFF_DURATION) $(HELM_OVERRIDES)
+	helm install $(DEPLOY_WAIT) -n $(NAMESPACE) --create-namespace $(HELM_RELEASE_NAME) $(OPERATOR_CHART) --set image.repo=null --set image.name=${OPERATOR_IMG} --set image.pullPolicy=$(HELM_IMAGE_PULL_POLICY) --set imagePullSecrets[0].name=priv-reg-cred $(HELM_OVERRIDES) --set controllers.scope=$(CONTROLLERS_SCOPE) --set controllers.vdbMaxBackoffDuration=$(VDB_MAX_BACKOFF_DURATION)


Suggested change

helm install $(DEPLOY_WAIT) -n $(NAMESPACE) --create-namespace $(HELM_RELEASE_NAME) $(OPERATOR_CHART) --set image.repo=null --set image.name=${OPERATOR_IMG} --set image.pullPolicy=$(HELM_IMAGE_PULL_POLICY) --set imagePullSecrets[0].name=priv-reg-cred $(HELM_OVERRIDES) --set controllers.scope=$(CONTROLLERS_SCOPE) --set controllers.vdbMaxBackoffDuration=$(VDB_MAX_BACKOFF_DURATION)

helm install $(DEPLOY_WAIT) -n $(NAMESPACE) --create-namespace $(HELM_RELEASE_NAME) $(OPERATOR_CHART) --set image.repo=null --set image.name=${OPERATOR_IMG} --set image.pullPolicy=$(HELM_IMAGE_PULL_POLICY) --set imagePullSecrets[0].name=priv-reg-cred --set controllers.scope=$(CONTROLLERS_SCOPE) --set controllers.vdbMaxBackoffDuration=$(VDB_MAX_BACKOFF_DURATION) $(HELM_OVERRIDES)

roypaulin · 2025-01-02T15:59:41Z

scripts/deploy-prometheus.sh

+NAMESPACE=''
+USERNAME=''
+PASSWORD=''
+DBNAME=''


Suggested change

DBNAME=''

VDBNAME=''

roypaulin · 2025-01-02T16:00:25Z

scripts/deploy-prometheus.sh

+INTERVAL='5s'
+
+function usage() {
+    echo "usage: $(basename $0) [-n <namespace>] [-l <label>] [-a <action>] [-u <username>] [-p <password>] [-d <dbname>] [-i <interval>]"


replace dbname with vdbname, they are not the same.

roypaulin · 2025-01-02T16:05:49Z

scripts/deploy-prometheus.sh

+  cat <<EOF | kubectl delete -f -
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: k8s-vertica-prometheus-$DBNAME
+  namespace: $NAMESPACE
+  labels:
+    release: $LABEL
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/instance: $DBNAME
+  namespaceSelector:
+    matchNames:
+      - $NAMESPACE
+  endpoints:
+    - basicAuth:
+        password:
+          key: password
+          name: prometheus-$DBNAME
+        username: 
+          key: username
+          name: prometheus-$DBNAME
+          optional: true
+      interval: $INTERVAL
+      path: /v1/metrics
+      port: vertica-http
+      scheme: https
+      tlsConfig:
+        insecureSkipVerify: true
+EOF


Suggested change

cat <<EOF | kubectl delete -f -

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: k8s-vertica-prometheus-$DBNAME

namespace: $NAMESPACE

labels:

release: $LABEL

spec:

selector:

matchLabels:

app.kubernetes.io/instance: $DBNAME

namespaceSelector:

matchNames:

- $NAMESPACE

endpoints:

- basicAuth:

password:

key: password

name: prometheus-$DBNAME

username:

key: username

name: prometheus-$DBNAME

optional: true

interval: $INTERVAL

path: /v1/metrics

port: vertica-http

scheme: https

tlsConfig:

insecureSkipVerify: true

EOF

kubectl delete servicemonitor k8s-vertica-prometheus-$DBNAME -n $NAMESPACE

roypaulin · 2025-01-02T16:07:13Z

scripts/deploy-prometheus.sh

+  cat <<EOF | kubectl delete -f -
+apiVersion: v1
+kind: Secret
+metadata:
+  namespace: $NAMESPACE
+  name: prometheus-$DBNAME
+data:
+  username: '$(echo -n $USERNAME | base64)'
+  password: '$(echo -n $PASSWORD | base64)'
+type: Opaque
+EOF


Suggested change

cat <<EOF | kubectl delete -f -

apiVersion: v1

kind: Secret

metadata:

namespace: $NAMESPACE

name: prometheus-$DBNAME

data:

username: '$(echo -n $USERNAME | base64)'

password: '$(echo -n $PASSWORD | base64)'

type: Opaque

EOF

kubectl delete secret prometheus-$DBNAME -n $NAMESPACE

roypaulin · 2025-01-02T16:11:33Z

scripts/deploy-prometheus.sh

+  namespace: $NAMESPACE
+  name: prometheus-$DBNAME


Suggested change

namespace: $NAMESPACE

name: prometheus-$DBNAME

namespace: $NAMESPACE

name: prometheus-$DBNAME

labels:

release: $LABEL

HaoYang0000 added 7 commits December 24, 2024 11:14

sample servicemonitor and values

b9e2bc9

update make file

f39bfb2

update sample service monitor

1c25379

update example values

6cf5e69

create script

dd38f92

update make file

b8b2110

remove template file

1682e71

HaoYang0000 marked this pull request as ready for review December 26, 2024 08:28

HaoYang0000 requested review from roypaulin, cchen-vertica, fenic-fawkes, qindotguan and LiboYu2 as code owners December 26, 2024 08:28

HaoYang0000 changed the title ~~[WIP]Deploy prometheus~~ Deploy prometheus Dec 26, 2024

remove unused content

5c3da03

roypaulin reviewed Dec 26, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

scripts/deploy-prometheus.sh Outdated Show resolved Hide resolved

scripts/deploy-prometheus.sh Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

HaoYang0000 added 2 commits December 26, 2024 11:59

address comments

4cd1754

add interval in scrpits and fix bug in namespace

593441a

HaoYang0000 added 3 commits December 26, 2024 12:26

add namespace in undeploy

b83799f

add vdb namespace customize

5d8c464

revert change

b0b6b88

roypaulin reviewed Dec 31, 2024

View reviewed changes

HaoYang0000 added 5 commits January 2, 2025 06:47

revert make file changes

aabad14

update default values

1006b9a

move values file to prometheus directory

e878ad6

update file path

5e260c8

add e2e test

825cbbc

roypaulin reviewed Jan 2, 2025

View reviewed changes

tests/e2e-leg-11/prometheustest/20-wait-for-data-pull.yaml Outdated Show resolved Hide resolved

tests/e2e-leg-11/prometheustest/10-deploy-prometheus.yaml Outdated Show resolved Hide resolved

address comments

d1a9398

roypaulin reviewed Jan 2, 2025

View reviewed changes

tests/e2e-leg-11/prometheustest/15-deploy-prometheus-service-monitor.yaml

Copy link

Collaborator

roypaulin Jan 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add an assert to check the service monitor

roypaulin reviewed Jan 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy prometheus #1013

Deploy prometheus #1013

HaoYang0000 commented Dec 24, 2024 •

edited

Loading

roypaulin left a comment

HaoYang0000 commented Dec 26, 2024

roypaulin left a comment

roypaulin Dec 31, 2024

HaoYang0000 Jan 2, 2025

roypaulin Jan 2, 2025

roypaulin Jan 2, 2025

roypaulin left a comment

roypaulin Jan 2, 2025

roypaulin Jan 2, 2025

roypaulin Jan 2, 2025

roypaulin Jan 2, 2025

roypaulin Jan 2, 2025

roypaulin Jan 2, 2025

	helm install $(DEPLOY_WAIT) -n $(NAMESPACE) --create-namespace $(HELM_RELEASE_NAME) $(OPERATOR_CHART) --set image.repo=null --set image.name=${OPERATOR_IMG} --set image.pullPolicy=$(HELM_IMAGE_PULL_POLICY) --set imagePullSecrets[0].name=priv-reg-cred $(HELM_OVERRIDES) --set controllers.scope=$(CONTROLLERS_SCOPE) --set controllers.vdbMaxBackoffDuration=$(VDB_MAX_BACKOFF_DURATION)
	helm install $(DEPLOY_WAIT) -n $(NAMESPACE) --create-namespace $(HELM_RELEASE_NAME) $(OPERATOR_CHART) --set image.repo=null --set image.name=${OPERATOR_IMG} --set image.pullPolicy=$(HELM_IMAGE_PULL_POLICY) --set imagePullSecrets[0].name=priv-reg-cred --set controllers.scope=$(CONTROLLERS_SCOPE) --set controllers.vdbMaxBackoffDuration=$(VDB_MAX_BACKOFF_DURATION) $(HELM_OVERRIDES)

Deploy prometheus #1013

Are you sure you want to change the base?

Deploy prometheus #1013

Conversation

HaoYang0000 commented Dec 24, 2024 • edited Loading

roypaulin left a comment

Choose a reason for hiding this comment

HaoYang0000 commented Dec 26, 2024

roypaulin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roypaulin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HaoYang0000 commented Dec 24, 2024 •

edited

Loading