Skip to content

Commit

Permalink
feat(alerts): CA cert creation via cert-manager (#637)
Browse files Browse the repository at this point in the history
* feat(alerts): CA cert creation via cert-manager

fixes #546

Helm `genCA` function led to reconciliation loops in Greenhouse.

Signed-off-by: Richard Tief <[email protected]>

* fix(alerts): correct formatting in values.yaml for caCert comment

Signed-off-by: IB Akshay <[email protected]>

* chore(alerts): improve description in values.yaml

Signed-off-by: Richard Tief <[email protected]>

* chore(alerts): disable certManager for helm-lint and helm-test

Signed-off-by: Richard Tief <[email protected]>

* chore(alerts): move ci folder into charts/

Signed-off-by: Richard Tief <[email protected]>

* chore(alerts): increase versions

Signed-off-by: Richard Tief <[email protected]>

* chore(alerts): name cert-manager resources like the organisation

Signed-off-by: Richard Tief <[email protected]>

* chore(alerts): add shared prometheus cert

For consecutive Prometheus installations.

Signed-off-by: Richard Tief <[email protected]>

* docs(alerts): adds cert-manager option values and explaination

Signed-off-by: Richard Tief <[email protected]>

* chore(alerts): add cert-manager options to plugindefinition

Signed-off-by: Richard Tief <[email protected]>

* docs(alerts): improve wording

Co-authored-by: Akshay Iyyadurai Balasundaram <[email protected]>

---------

Signed-off-by: Richard Tief <[email protected]>
Signed-off-by: IB Akshay <[email protected]>
Co-authored-by: IB Akshay <[email protected]>
  • Loading branch information
richardtief and ibakshay authored Feb 13, 2025
1 parent 39f7000 commit 387cbea
Show file tree
Hide file tree
Showing 13 changed files with 160 additions and 191 deletions.
26 changes: 12 additions & 14 deletions alerts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,10 @@ Greenhouse regularly performs integration tests that are bundled with **alerts**

| Name | Description | Value |
| ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ------- |
| `global.caCert` | Additional caCert to add to the CA bundle | `""` |
| `alerts.commonLabels` | Labels to apply to all resources | `{}` |
| `alerts.defaultRules.create` | Creates community Alertmanager alert rules. | `true` |
| `alerts.defaultRules.labels` | kube-monitoring `plugin: <plugin.name>` to evaluate Alertmanager rules. | `{}` |
| `alerts.alertmanager.enabled` | Deploy Prometheus Alertmanager | `true` |
| `alerts.alertmanager.annotations` | Annotations for Alertmanager | `{}` |
| `alerts.alertmanager.config` | Alertmanager configuration directives. | `{}` |
Expand All @@ -76,18 +79,15 @@ Greenhouse regularly performs integration tests that are bundled with **alerts**
| `alerts.alertmanager.alertmanagerConfig.webhook.routes[].name` | Name of the webhook route. | `""` |
| `alerts.alertmanager.alertmanagerConfig.webhook.routes[].url` | Webhook url to post alerts to. | `""` |
| `alerts.alertmanager.alertmanagerConfig.webhook.routes[].matchers` | List of matchers that the alert's label should match. matchType <string>, name <string>, regex <boolean>, value <string> | `[]` |
| `alerts.alertmanager.alertmanagerSpec.alertmanagerConfiguration` | AlermanagerConfig to be used as top level configuration | `false` |
| `alerts.alertmanager.alertmanagerConfig.webhook.routes[].matchers` | List of matchers that the alert's label should match. matchType <string>, name <string>, regex <boolean>, value <string> | `[]` |

| `alerts.auth.secretName` | Use custom secret for Alertmanager authentication | `""` |
| `alerts.auth.autoGenerateCert.enabled` | TLS Certificate Option 1: Use Helm to automatically generate self-signed certificate. | `true` |
| `alerts.auth.autoGenerateCert.recreate` | If set to true, new key/certificate is generated on Helm upgrade. | `false` |
| `alerts.auth.autoGenerateCert.certPeriodDays` | Cert period time in days. The default is 365 days. | `365` |
| `alerts.auth.certFile` | Path to your own PEM-encoded certificate. | `""` |
| `alerts.auth.keyFile` | Path to your own PEM-encoded private key. | `""` |
| `alerts.auth.caFile` | Path to CA cert. | `""` |
### cert-manager options

| `alerts.defaultRules.create` | Creates community Alertmanager alert rules. | `true` |
| `alerts.defaultRules.labels` | kube-monitoring `plugin: <plugin.name>` to evaluate Alertmanager rules. | `{}` |
| `alerts.alertmanager.alertmanagerSpec.alertmanagerConfiguration` | AlermanagerConfig to be used as top level configuration | `false` |
| `alerts.certManager.enabled` | Creates `jetstack/cert-manager` resources to generate Issuer and Certificates for Prometheus authentication. | `true` |
| `alerts.certManager.rootCert.duration` | Duration, how long the root certificate is valid. | `"5y"` |
| `alerts.certManager.admissionCert.duration` | Duration, how long the admission certificate is valid. | `"1y"` |
| `alerts.certManager.issuerRef.name` | Name of the existing Issuer to use. | `""` |

### Supernova options

Expand Down Expand Up @@ -216,11 +216,9 @@ alertmanagerConfiguration:

## TLS Certificate Requirement

Greenhouse onboarded Prometheus installations need to communicate with the Alertmanager component to enable advanced processing of alerts. The Alertmanager Ingress requires a TLS certificate to be configured and trusted by Prometheus to ensure the communication. There are various ways in which you can generate/configure the required TLS certificate.
Greenhouse onboarded Prometheus installations need to communicate with the Alertmanager component to enable processing of alerts. If an Alertmanager Ingress is enabled, this requires a TLS certificate to be configured and trusted by Alertmanger to ensure the communication. To enable automatic self-signed TLS certificate provisioning via cert-manager, set the `alerts.certManager.enabled` value to `true`.

- You can use an automatically generated self-signed certificate by setting `alerts.auth.autoGenerateCert.enabled` to `true`. Helm will create a self-signed cert and a secret for you.
- You can use your own generated self-signed certificate by setting `alerts.auth.autoGenerateCert.enabled` to `false`. You should provide the necessary values to `alerts.auth.certFile`, `alerts.auth.keyFile`, and `alerts.auth.caFile`.
- You can also sideload custom certificate by disabling `alerts.auth.autoGenerateCert.enabled` to `false` while setting your custom cert secret name in `alerts.auth.secretName`
Note: Prerequisite of this feature is a installed [jetstack/cert-manager](https://github.com/jetstack/cert-manager) which can be implemented via the Greenhouse [cert-manager](https://github.com/cloudoperators/greenhouse-extensions/tree/main/cert-manager) Plugin.

## Examples

Expand Down
2 changes: 1 addition & 1 deletion alerts/charts/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ maintainers:
name: alerts
sources:
- https://github.com/cloudoperators/greenhouse-extensions
version: 0.18.0
version: 0.19.0
keywords:
- prometheus-alertmanager
dependencies:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,5 @@ alerts:
enabled: true
hosts:
- dummy-host
certManager:
enabled: false
33 changes: 0 additions & 33 deletions alerts/charts/templates/_helper.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -47,36 +47,3 @@ plugin: {{ $root.Release.Name }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Return certificate and CA for Prometheus to push alerts.
It handles variants when a cert has to be generated by Helm,
a cert is loaded from an existing secret or is provided via `.Values`
*/}}
{{- define "alerts.generateCert" -}}
{{- $caCertEnc := "" }}
{{- $certCrtEnc := "" }}
{{- $certKeyEnc := "" }}
{{- if .Values.alerts.auth.autoGenerateCert.enabled }}
{{- $prevSecret := (lookup "v1" "Secret" .Release.Namespace (default (printf "%s-monitoring-ca" .Release.Namespace) .Values.alerts.auth.secretName )) }}
{{- if and (not .Values.alerts.auth.autoGenerateCert.recreate) $prevSecret }}
{{- $certCrtEnc = index $prevSecret "data" "tls.crt" }}
{{- $certKeyEnc = index $prevSecret "data" "tls.key" }}
{{- $caCertEnc = index $prevSecret "data" "ca.crt" }}
{{- else }}
{{- $altNames := list ( printf "%s-alertmanager.%s" .Release.Name .Release.Namespace ) ( printf "%s-alertmanager.%s.svc" .Release.Name .Release.Namespace ) -}}
{{- $tmpperioddays := int .Values.alerts.auth.autoGenerateCert.certPeriodDays | default 365 }}
{{- $ca := genCA "greenhouse-monitoring-ca" $tmpperioddays }}
{{- $cert := genSignedCert .Release.Name nil $altNames $tmpperioddays $ca }}
{{- $certCrtEnc = b64enc $cert.Cert }}
{{- $certKeyEnc = b64enc $cert.Key }}
{{- $caCertEnc = b64enc $ca.Cert }}
{{- end }}
{{- else }}
{{- $certCrtEnc = .Files.Get .Values.alerts.auth.certFile | b64enc }}
{{- $certKeyEnc = .Files.Get .Values.alerts.auth.keyFile | b64enc }}
{{- $caCertEnc = .Files.Get .Values.alerts.auth.caFile | b64enc }}
{{- end }}
{{- $result := dict "crt" $certCrtEnc "key" $certKeyEnc "ca" $caCertEnc }}
{{- $result | toYaml }}
{{- end }}
18 changes: 0 additions & 18 deletions alerts/charts/templates/am-ca-bundle-secret.yaml

This file was deleted.

23 changes: 0 additions & 23 deletions alerts/charts/templates/am-webhook-secrets.yaml

This file was deleted.

55 changes: 0 additions & 55 deletions alerts/charts/templates/ca-secret-issuer-cert.yaml

This file was deleted.

85 changes: 85 additions & 0 deletions alerts/charts/templates/certmanager.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
{{- if .Values.alerts.certManager.enabled -}}
{{- if not .Values.alerts.certManager.issuerRef -}}
# Create a selfsigned Issuer in order to create a root CA certificate for the org
# signing alertmanager serving certificates
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: {{ .Release.Namespace }}-prometheus-issuer
labels:
{{- include "kube-prometheus-stack.labels" . | indent 4 }}
spec:
selfSigned: {}
---
# Generate a CA Certificate used to sign certificates for the alertmanager
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{ .Release.Namespace }}-prometheus-root-cert
labels:
{{- include "kube-prometheus-stack.labels" . | indent 4 }}
spec:
secretName: {{ .Release.Namespace }}-prometheus-root-cert
duration: {{ .Values.alerts.certManager.rootCert.duration | default "43800h0m0s" | quote }}
issuerRef:
name: {{ .Release.Namespace }}-prometheus-issuer
commonName: "ca.prometheus.greenhouse"
isCA: true
---
# Create an Issuer that uses the above generated CA certificate to issue certs
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: {{ .Release.Namespace }}-prometheus-root-issuer
labels:
{{- include "kube-prometheus-stack.labels" . | indent 4 }}
spec:
ca:
secretName: {{ .Release.Namespace }}-root-cert
{{- end }}
---
# generate a server certificate for the alertmanager to use
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{ include "kube-prometheus-stack.fullname" . }}-cert
labels:
{{- include "kube-prometheus-stack.labels" . | indent 4 }}
spec:
secretName: tls-{{ include "kube-prometheus-stack.fullname" . }}-cert
duration: {{ .Values.alerts.certManager.admissionCert.duration | default "8760h0m0s" | quote }}
issuerRef:
{{- if .Values.alerts.certManager.issuerRef }}
{{- toYaml .Values.alerts.certManager.issuerRef | nindent 4 }}
{{- else }}
name: {{ .Release.Namespace }}-root-issuer
{{- end }}
dnsNames:
- {{ include "kube-prometheus-stack.fullname" . }}
- {{ include "kube-prometheus-stack.fullname" . }}.{{ .Release.Namespace }}
- {{ include "kube-prometheus-stack.fullname" . }}.{{ .Release.Namespace }}.svc
{{- if and .Values.alerts.alertmanager.ingress.enabled .Values.alerts.alertmanager.ingress.hosts }}
{{ toYaml .Values.alerts.alertmanager.ingress.hosts | indent 2 }}
{{- end }}
---
# generate a shared client certificate for prometheus to use
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: prometheus-{{ .Release.Namespace }}-cert
labels:
{{- include "kube-prometheus-stack.labels" . | indent 4 }}
spec:
secretName: tls-prometheus-{{ .Release.Namespace }}
duration: {{ .Values.alerts.certManager.admissionCert.duration | default "8760h0m0s" | quote }}
issuerRef:
{{- if .Values.alerts.certManager.issuerRef }}
{{- toYaml .Values.alerts.certManager.issuerRef | nindent 4 }}
{{- else }}
name: {{ .Release.Namespace }}-root-issuer
{{- end }}
dnsNames:
{{- if .Values.global.greenhouse.baseDomain }}
- {{ printf "*.s%" .Values.global.greenhouse.baseDomain }}
{{- end }}
{{- end -}}
19 changes: 19 additions & 0 deletions alerts/charts/templates/hooks/caBundle-secret.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{{- if and .Values.alerts.alertmanager.enabled .Values.alerts.alertmanager.ingress.enabled .Values.global.caCert }}
{{- $extCABundle := .Values.global.caCert }}
{{- $intCASecret := (lookup "v1" "Secret" $.Release.Namespace (printf "%s-%s" (include "kube-prometheus-stack.fullname") "-root-cert")).data }}
{{- $intCABundle := get $intCASecret "ca.crt" | b64dec }}
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: {{ $.Release.Namespace }}-ca-bundle
labels:
{{- include "kube-prometheus-stack.labels" . | indent 4 }}
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
## Ensure this is run before release installation and upgrade
"helm.sh/hook-weight": "-5"
data:
ca.crt: {{ printf "%s%s" $extCABundle $intCABundle | b64enc | quote }}
{{- end }}
8 changes: 4 additions & 4 deletions alerts/charts/templates/tests/test-alerts-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ data:
}
{{- end }}
{{- if and .Values.alerts.auth.autoGenerateCert.enabled .Values.alerts.alertmanager.ingress.enabled .Values.alerts.alertmanager.ingress.hosts (.Capabilities.APIVersions.Has "cert-manager.io/v1") }}
@test "Verify certificate creation and validation against the API endpoint of the alert manager" {
{{- if and .Values.alerts.alertmanager.ingress.enabled .Values.alerts.alertmanager.ingress.hosts .Values.alerts.certManager.enabled }}
@test "Generated server certificate for the Prometheus Alertmanager to use" {
verify "there is 1 issuer named '{{ .Release.Namespace }}-monitoring-issuer'"
verify "there is 1 certificate named '{{ .Release.Namespace }}-prometheus-auth'"
verify "there is 1 secret named 'tls-{{ .Release.Namespace }}-prometheus-auth'"
verify "there is 1 certificate named '{{ include "kube-prometheus-stack.fullname" . }}-cert'"
verify "there is 1 secret named '{{ include "kube-prometheus-stack.fullname" . }}-cert'"
url="https://{{ first .Values.alerts.alertmanager.ingress.hosts }}/-/healthy"
Expand Down
8 changes: 4 additions & 4 deletions alerts/charts/templates/tests/test-alerts.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,20 @@ spec:
- name: tests
mountPath: /tests
readOnly: true
{{- if and .Values.alerts.auth.autoGenerateCert.enabled (.Capabilities.APIVersions.Has "cert-manager.io/v1") }}
{{- if and .Values.alerts.alertmanager.ingress.enabled .Values.alerts.alertmanager.ingress.hosts .Values.alerts.certManager.enabled }}
- name: tls-cert
mountPath: /tls-assets
readOnly: true
{{- end }}
{{- end }}
volumes:
- name: tests
configMap:
name: {{ .Release.Name }}-test
{{- if and .Values.alerts.auth.autoGenerateCert.enabled (.Capabilities.APIVersions.Has "cert-manager.io/v1") }}
{{- if and .Values.alerts.alertmanager.ingress.enabled .Values.alerts.alertmanager.ingress.hosts .Values.alerts.certManager.enabled }}
- name: tls-cert
secret:
defaultMode: 420
secretName: {{ print "tls-%s-prometheus-auth" $.Release.Namespace }}
{{- end }}
{{- end }}
restartPolicy: Never
{{- end -}}
Loading

0 comments on commit 387cbea

Please sign in to comment.