CLIP-1826: Add user provided certificates to the default Java trustst…

…ore (#663)

* Add certs to Java trust store

* Add docs

---------

Co-authored-by: Yevhen Ivantsov <[email protected]>

docs/docs/userguide/CONFIGURATION.md

@@ -519,3 +519,53 @@ readinessProbe:
 !!!warning "`startupProbe` and `livenessProbe`"
 
     Both `startupProbe` and `livenessProbe` are disabled by default. Make sure you go through the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/){.external} before enabling such probes. Misconfiguration can result in unwanted container restarts and failed "cold" starts.
+
+## :material-certificate: Self Signed Certificates
+
+Accessing applications or websites that are encrypted with SSL using certificates not signed by a public authority will result in a connection error. The stacktrace will contain the following:
+
+```shell
+caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
+```
+
+To fix the issue, self signed certificates need to be added to Java truststore. First off, you need to create a [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/){.external} containing base64-encoded certificate(s). Here's an example [kubectl command](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/#use-source-files){.external} to create a secret from 2 local files:
+
+```shell
+kubectl create secret generic dev-certificates \
+    --from-file=stg.crt=./stg.crt \
+    --from-file=dev.crt=./dev.crt -n $namespace
+```
+
+The resulting secret will have the following data:
+
+```yaml
+data:
+  stg.crt: base64encodedstgcrt
+  dev.crt: base64encodeddevcrt
+```
+
+!!!info "You can have as many keys (certificates) in the secret as required. All keys will be mounted as files to `/tmp/crt` in the container and imported into Java truststore. In the example above, certificates will be mounted as `/tmp/crt/stg.crt` and `/tmp/crt/dev.crt`. File extension in the secret keys does not matter as long as the file is a valid certificate."
+
+Once done, provide the secret name in Helm values: 
+
+```yaml
+jira:
+  additionalCertificates:
+     secretName: dev-certificates
+```
+
+Helm chart will add additional `volumeMounts` and `volumes` to the pod(s), as well as an extra init container that will:
+
+* copy the default Java cacerts to a runtime volume shared between the init container and the main container at `/var/ssl`
+* run [keytool -import](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html){.external} to import all certificates in `/tmp/crt` mounted from `dev-certificates` secret to `/var/ssl/cacerts`
+
+`-Djavax.net.ssl.trustStore=/var/ssl/cacerts` system property will be automatically added to `JVM_SUPPORT_RECOMMENDED_ARGS` environment variable.
+
+If necessary, it is possible to override the default `keytool -import` command:
+
+```yaml
+jira:
+  additionalCertificates:
+     secretName: dev-certificates
+     customCmd: keytool -import ...
+```

src/main/charts/bamboo/README.md

@@ -32,6 +32,7 @@ Kubernetes: `>=1.21.x-0`
 | bamboo.accessLog.localHomeSubPath | string | `"log"` | The subdirectory within the local-home volume where access logs should be stored.  |
 | bamboo.accessLog.mountPath | string | `"/opt/atlassian/bamboo/logs"` | The path within the Bamboo container where the local-home volume should be mounted in order to capture access logs.  |
 | bamboo.additionalBundledPlugins | list | `[]` | Specifies a list of additional Bamboo plugins that should be added to the Bamboo container. Note plugins installed via this method will appear as bundled plugins rather than user plugins. These should be specified in the same manner as the 'additionalLibraries' property. Additional details: https://atlassian.github.io/data-center-helm-charts/examples/external_libraries/EXTERNAL_LIBS/  NOTE: only .jar files can be loaded using this approach. OBR's can be extracted (unzipped) to access the associated .jar  An alternative to this method is to install the plugins via "Manage Apps" in the product system administration UI.  |
+| bamboo.additionalCertificates | object | `{"customCmd":null,"secretName":null}` | Certificates to be added to Java truststore. Provide reference to a secret that contains the certificates  |
 | bamboo.additionalEnvironmentVariables | list | `[]` | Defines any additional environment variables to be passed to the Bamboo container. See https://hub.docker.com/r/atlassian/bamboo-server for supported variables.  |
 | bamboo.additionalJvmArgs | list | `[]` | Specifies a list of additional arguments that can be passed to the Bamboo JVM, e.g. system properties.  |
 | bamboo.additionalLibraries | list | `[]` | Specifies a list of additional Java libraries that should be added to the Bamboo container. Each item in the list should specify the name of the volume that contains the library, as well as the name of the library file within that volume's root directory. Optionally, a subDirectory field can be included to specify which directory in the volume contains the library file. Additional details: https://atlassian.github.io/data-center-helm-charts/examples/external_libraries/EXTERNAL_LIBS/  |

src/main/charts/bamboo/templates/_helpers.tpl

@@ -111,6 +111,10 @@ on Tomcat's logs directory. THis ensures that Tomcat+Bamboo logs get captured in
   {{- if .Values.volumes.sharedHome.subPath }}
   subPath: {{ .Values.volumes.sharedHome.subPath | quote }}
   {{- end }}
+{{- if .Values.bamboo.additionalCertificates.secretName }}
+- name: keystore
+  mountPath: /var/ssl
+{{- end }}
 {{- end }}
 
 {{/*
@@ -205,6 +209,13 @@ For each additional plugin declared, generate a volume mount that injects that l
 {{- with .Values.volumes.additional }}
 {{- toYaml . | nindent 0 }}
 {{- end }}
+{{- if .Values.bamboo.additionalCertificates.secretName }}
+- name: keystore
+  emptyDir: {}
+- name: certs
+  secret:
+    secretName: {{ .Values.bamboo.additionalCertificates.secretName }}
+{{- end }}
 {{- end }}
 
 {{- define "bamboo.volumes.localHome" -}}
@@ -307,3 +318,11 @@ Define additional hosts here to allow template overrides when used as a sub char
 {{- toYaml . }}
 {{- end }}
 {{- end }}
+
+{{- define "bamboo.addCrtToKeystoreCmd" }}
+{{- if .Values.bamboo.additionalCertificates.customCmd}}
+{{ .Values.bamboo.additionalCertificates.customCmd}}
+{{- else }}
+set -e; for crt in /tmp/crt/*.*; do echo "Adding $crt to keystore"; keytool -import -cacerts -storepass changeit -noprompt -alias $(echo $(basename $crt)) -file $crt; done; cp $JAVA_HOME/lib/security/cacerts /var/ssl/cacerts
+{{- end }}
+{{- end }}

src/main/charts/bamboo/templates/config-jvm.yaml

@@ -10,6 +10,9 @@ data:
     {{ . }}
     {{- end }}
     -XX:ActiveProcessorCount={{ include "flooredCPU" .Values.bamboo.resources.container.requests.cpu }}
+    {{- if .Values.bamboo.additionalCertificates.secretName }}
+    -Djavax.net.ssl.trustStore=/var/ssl/cacerts
+    {{- end }}
     {{ include "common.jmx.javaagent" . | indent 4 | trim }}
   max_heap: {{ .Values.bamboo.resources.jvm.maxHeap }}
   min_heap: {{ .Values.bamboo.resources.jvm.minHeap }}

src/main/charts/bamboo/templates/statefulset.yaml

@@ -55,6 +55,18 @@ spec:
           command: ["sh", "-c", {{ include "bamboo.sharedHome.permissionFix.command" . | quote }}]
         {{- end }}
         {{- include "common.jmx.initContainer" . | nindent 8 }}
+        {{- if .Values.bamboo.additionalCertificates.secretName }}
+        - name: import-certs
+          image: {{ include "bamboo.image" . | quote }}
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          volumeMounts:
+            - name: keystore
+              mountPath: /var/ssl
+            - name: certs
+              mountPath: /tmp/crt
+          command: ["/bin/bash"]
+          args: ["-c", {{ include "bamboo.addCrtToKeystoreCmd" . }}]
+        {{- end }}
       containers:
         - name: {{ if .Values.bamboo.useHelmReleaseNameAsContainerName}}{{ include "common.names.fullname" . }}{{ else }}{{ .Chart.Name }}{{ end }}
           image: {{ include "bamboo.image" . | quote }}

src/main/charts/bamboo/values.yaml

@@ -829,6 +829,12 @@ bamboo:
   #   labelSelector:
   #     matchLabels:
 
+  # -- Certificates to be added to Java truststore. Provide reference to a secret that contains the certificates
+  #
+  additionalCertificates:
+    secretName:
+    customCmd:
+
 # Monitoring
 #
 monitoring:

src/main/charts/bitbucket/README.md

@@ -31,6 +31,7 @@ Kubernetes: `>=1.21.x-0`
 | additionalLabels | object | `{}` | Additional labels that should be applied to all resources  |
 | affinity | object | `{}` | Standard Kubernetes affinities that will be applied to all Bitbucket pods Due to the performance requirements it is highly recommended running all Bitbucket pods in the same availability zone as your dedicated NFS server. To achieve this, you can define `affinity` and `podAffinity` rules that will place all pods into the same zone, and therefore minimise the real distance between the application pods and the shared storage. More specific documentation can be found in the official Affinity and Anti-affinity documentation:  https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity  This is an example on how to ensure the pods are in the same zone as NFS server that is labeled with `role=nfs-server`:    podAffinity:    requiredDuringSchedulingIgnoredDuringExecution:      - labelSelector:          matchExpressions:            - key: role              operator: In              values:                - nfs-server # needs to be the same value as NFS server deployment        topologyKey: topology.kubernetes.io/zone |
 | bitbucket.additionalBundledPlugins | list | `[]` | Specifies a list of additional Bitbucket plugins that should be added to the Bitbucket container. Note plugins installed via this method will appear as bundled plugins rather than user plugins. These should be specified in the same manner as the 'additionalLibraries' property. Additional details: https://atlassian.github.io/data-center-helm-charts/examples/external_libraries/EXTERNAL_LIBS/  NOTE: only .jar files can be loaded using this approach. OBR's can be extracted (unzipped) to access the associated .jar  An alternative to this method is to install the plugins via "Manage Apps" in the product system administration UI.  |
+| bitbucket.additionalCertificates | object | `{"customCmd":null,"secretName":null}` | Certificates to be added to Java truststore. Provide reference to a secret that contains the certificates  |
 | bitbucket.additionalEnvironmentVariables | list | `[]` | Defines any additional environment variables to be passed to the Bitbucket container. See https://hub.docker.com/r/atlassian/bitbucket-server for supported variables.  |
 | bitbucket.additionalJvmArgs | list | `[]` | Specifies a list of additional arguments that can be passed to the Bitbucket JVM, e.g. system properties.  |
 | bitbucket.additionalLibraries | list | `[]` | Specifies a list of additional Java libraries that should be added to the Bitbucket container. Each item in the list should specify the name of the volume that contains the library, as well as the name of the library file within that volume's root directory. Optionally, a subDirectory field can be included to specify which directory in the volume contains the library file. Additional details: https://atlassian.github.io/data-center-helm-charts/examples/external_libraries/EXTERNAL_LIBS/  |
@@ -55,6 +56,7 @@ Kubernetes: `>=1.21.x-0`
 | bitbucket.livenessProbe.initialDelaySeconds | int | `60` | Time to wait before starting the first probe  |
 | bitbucket.livenessProbe.periodSeconds | int | `5` | How often (in seconds) the Bitbucket container liveness probe will run  |
 | bitbucket.livenessProbe.timeoutSeconds | int | `1` | Number of seconds after which the probe times out  |
+| bitbucket.mesh.additionalCertificates | object | `{"customCmd":null,"secretName":null}` | Certificates to be added to Java truststore. Provide reference to a secret that contains the certificates  |
 | bitbucket.mesh.additionalEnvironmentVariables | object | `{}` | Defines any additional environment variables to be passed to the Bitbucket mesh containers.  |
 | bitbucket.mesh.additionalFiles | string | `nil` | Additional existing ConfigMaps and Secrets not managed by Helm that should be mounted into service container  |
 | bitbucket.mesh.additionalInitContainers | object | `{}` | Additional initContainer definitions that will be added to all Bitbucket pods  |

src/main/charts/bitbucket/templates/_helpers.tpl

@@ -254,6 +254,13 @@ Define additional hosts here to allow template overrides when used as a sub char
 {{- with .Values.volumes.additional }}
 {{- toYaml . | nindent 0 }}
 {{- end }}
+{{- if .Values.bitbucket.additionalCertificates.secretName }}
+- name: keystore
+  emptyDir: {}
+- name: certs
+  secret:
+    secretName: {{ .Values.bitbucket.additionalCertificates.secretName }}
+{{- end }}
 {{- end }}
 
 {{- define "bitbucket.volumes.localHome" -}}
@@ -445,3 +452,19 @@ volumeClaimTemplates:
     {{- . }}
     {{- end }}
 {{- end}}
+
+{{- define "bitbucket.addCrtToKeystoreCmd" }}
+{{- if .Values.bitbucket.additionalCertificates.customCmd}}
+{{ .Values.bitbucket.additionalCertificates.customCmd}}
+{{- else }}
+set -e; for crt in /tmp/crt/*.*; do echo "Adding $crt to keystore"; keytool -import -cacerts -storepass changeit -noprompt -alias $(echo $(basename $crt)) -file $crt; done; cp $JAVA_HOME/lib/security/cacerts /var/ssl/cacerts
+{{- end }}
+{{- end }}
+
+{{- define "bitbucketMesh.addCrtToKeystoreCmd" }}
+{{- if .Values.bitbucket.mesh.additionalCertificates.customCmd}}
+{{ .Values.bitbucket.mesh.additionalCertificates.customCmd}}
+{{- else }}
+set -e; for crt in /tmp/crt/*.*; do echo "Adding $crt to keystore"; keytool -import -cacerts -storepass changeit -noprompt -alias $(echo $(basename $crt)) -file $crt; done; cp $JAVA_HOME/lib/security/cacerts /var/ssl/cacerts
+{{- end }}
+{{- end }}

src/main/charts/bitbucket/templates/config-jvm-mesh.yaml

@@ -13,5 +13,8 @@ data:
     {{- if .Values.monitoring.exposeJmxMetrics }}
     -javaagent:{{ .Values.monitoring.jmxExporterCustomJarLocation | default (printf "%s/jmx_prometheus_javaagent.jar" ( .Values.bitbucket.mesh.volume.mountPath)) }}={{ .Values.monitoring.jmxExporterPort}}:/opt/atlassian/jmx/jmx-config.yaml
     {{- end }}
+    {{- if .Values.bitbucket.mesh.additionalCertificates.secretName }}
+    -Djavax.net.ssl.trustStore=/var/ssl/cacerts
+    {{- end }}
   max_heap: {{ .Values.bitbucket.mesh.resources.jvm.maxHeap }}
   min_heap: {{ .Values.bitbucket.mesh.resources.jvm.minHeap }}

src/main/charts/bitbucket/templates/config-jvm.yaml

@@ -14,5 +14,8 @@ data:
     {{- if .Values.monitoring.exposeJmxMetrics }}
     -Dplugin.bitbucket-git.mesh.sidecar.jvmArgs=-javaagent:{{ .Values.monitoring.jmxExporterCustomJarLocation | default (printf "%s/jmx_prometheus_javaagent.jar"  .Values.volumes.sharedHome.mountPath) }}=9998:/opt/atlassian/jmx/jmx-config.yaml
     {{- end }}
+    {{- if .Values.bitbucket.additionalCertificates.secretName }}
+    -Djavax.net.ssl.trustStore=/var/ssl/cacerts
+    {{- end }}
   max_heap: {{ .Values.bitbucket.resources.jvm.maxHeap }}
   min_heap: {{ .Values.bitbucket.resources.jvm.minHeap }}

src/main/charts/bitbucket/templates/statefulset-mesh.yaml

@@ -47,6 +47,18 @@ spec:
           securityContext:
             runAsUser: 0 # make sure we run as root to avoid permission denied
         {{- end }}
+        {{- if .Values.bitbucket.mesh.additionalCertificates.secretName }}
+        - name: import-certs
+          image: {{ .Values.bitbucket.mesh.image.repository }}:{{ .Values.bitbucket.mesh.image.tag }}
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          volumeMounts:
+            - name: keystore
+              mountPath: /var/ssl
+            - name: certs
+              mountPath: /tmp/crt
+          command: ["/bin/bash"]
+          args: ["-c", {{ include "bitbucketMesh.addCrtToKeystoreCmd" . }}]
+        {{- end }}
       containers:
         - name: {{ if .Values.bitbucket.useHelmReleaseNameAsContainerName}}{{ include "common.names.fullname" . }}-mesh{{ else }}{{ .Chart.Name }}-mesh{{ end }}
           image: {{ .Values.bitbucket.mesh.image.repository }}:{{ .Values.bitbucket.mesh.image.tag }}
@@ -74,6 +86,10 @@ spec:
               mountPath: {{ .mountPath }}/{{ .key }}
               subPath: {{ .key }}
             {{ end }}
+            {{- if .Values.bitbucket.mesh.additionalCertificates.secretName }}
+            - name: keystore
+              mountPath: /var/ssl
+            {{- end }}
             {{- include "common.jmx.config.volumeMounts" . | nindent 12 }}
           {{- with .Values.bitbucket.mesh.resources.container }}
           resources:
@@ -147,6 +163,13 @@ spec:
               - key: {{ .key }}
                 path: {{ .key }}
         {{ end }}
+        {{- if .Values.bitbucket.mesh.additionalCertificates.secretName }}
+        - name: keystore
+          emptyDir: {}
+        - name: certs
+          secret:
+            secretName: {{ .Values.bitbucket.mesh.additionalCertificates.secretName }}
+        {{- end }}
         {{ include "common.jmx.config.volume" . | nindent 8 }}
   {{ include "bitbucket.mesh.volumeClaimTemplates" . | nindent 2 }}
 {{ end }}

src/main/charts/bitbucket/templates/statefulset.yaml

@@ -68,6 +68,18 @@ spec:
           command: ["sh", "-c", {{ include "bitbucket.sharedHome.permissionFix.command" . | quote }}]
         {{- end }}
         {{- include "common.jmx.initContainer" . | nindent 8 }}
+        {{- if .Values.bitbucket.additionalCertificates.secretName }}
+        - name: import-certs
+          image: {{ include "bitbucket.image" . | quote }}
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          volumeMounts:
+            - name: keystore
+              mountPath: /var/ssl
+            - name: certs
+              mountPath: /tmp/crt
+          command: ["/bin/bash"]
+          args: ["-c", {{ include "bitbucket.addCrtToKeystoreCmd" . }}]
+        {{- end }}
       containers:
         - name: {{ if .Values.bitbucket.useHelmReleaseNameAsContainerName}}{{ include "common.names.fullname" . }}{{ else }}{{ .Chart.Name }}{{ end }}
           image: {{ include "bitbucket.image" . | quote }}
@@ -131,6 +143,10 @@ spec:
               subPath: {{ .Values.volumes.sharedHome.subPath | quote }}
               {{- end }}
             {{- end }}
+            {{- if .Values.bitbucket.additionalCertificates.secretName }}
+            - name: keystore
+              mountPath: /var/ssl
+            {{- end }}
             {{- include "bitbucket.additionalVolumeMounts" . | nindent 12 }}
             {{- include "common.jmx.config.volumeMounts" . | nindent 12 }}
             {{- include "bitbucket.additionalLibraries" . | nindent 12 }}

src/main/charts/bitbucket/values.yaml

@@ -1028,6 +1028,12 @@ bitbucket:
       #
       terminationGracePeriodSeconds: 35
 
+    # -- Certificates to be added to Java truststore. Provide reference to a secret that contains the certificates
+    #
+    additionalCertificates:
+      secretName:
+      customCmd:
+
   # -- Specifies a list of additional arguments that can be passed to the Bitbucket JVM, e.g.
   # system properties.
   #
@@ -1102,6 +1108,12 @@ bitbucket:
   #   labelSelector:
   #     matchLabels:
 
+  # -- Certificates to be added to Java truststore. Provide reference to a secret that contains the certificates
+  #
+  additionalCertificates:
+    secretName:
+    customCmd:
+
 # Monitoring
 #
 monitoring: