Nuxeo Platform CI

Configuration as code for the Nuxeo Platform CI in Kubernetes.

Principles

Jenkins

Jenkins is installed with the Jenkins Helm chart.

The Jenkins Helm chart is deployed with Helmfile and configured with a set of custom values overriding the default ones, defined in the ./charts/jenkins/values*.yaml.gotmpl files.

This configuration mostly includes:

• Jenkins image
• Java options
• Jenkins plugins
• Jenkins Configuration as Code (JCasC), among which:
  • Authorization
  • Credentials
  • Jenkins and plugin configuration
  • Kubernetes Cloud configuration, including pod templates
  • Jobs, using the Jenkins Job DSL Plugin

When synchronizing the Kubernetes cluster with the resources from the helmfile, depending on the changes:

• The Jenkins pod will not be restarted if the changes only impact Jenkins Configuration as Code, thanks to the config-reload container that takes care of hot reloading the configuration. This includes pod templates and jobs!
• The Jenkins pod will be restarted if the changes impact anything else than Jenkins Configuration as Code, typically the Jenkins image, plugins or Java options.

Nexus

Nexus is used as:

• An internal Docker registry for the images built in the Jenkins pipelines.
• An internal Maven proxy repository to the main upstream repository.

Requirements

• Helm 3
• Helmfile
• Helm Diff plugin
• Kustomize

Installation

Kubernetes Cluster Initialization

Target Namespace

Define the target namespace variable:

NAMESPACE=target-namespace

Create the $NAMESPACE namespace:

kubectl create ns $NAMESPACE

Secrets

Import the required secrets from the platform namespace:

./secrets/import-secrets.sh ./secrets/secrets platform

Create the secret containing the credentials for the Jenkins Configuration as Code (JCasC):

(
cat << EOF
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: jenkins-casc
data:
  gitHubAppKey: ********
  gitHubOAuthClientId: ********
  gitHubOAuthSecret: ********
  gitHubToken: ********
  gitHubUser: ********
  jiraPassword: ********
  slackToken: ********
EOF
) | kubectl apply --namespace=$NAMESPACE -f -

Create the secret containing the Nexus credentials:

(
cat << EOF
apiVersion: v1
kind: Secret
type: Opaque
data:
  admin.password: ********
  admin.username: ********
metadata:
  annotations:
    meta.helm.sh/release-name: nexus
    meta.helm.sh/release-namespace: $NAMESPACE
  labels:
    app.kubernetes.io/instance: nexus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nexus
    app.kubernetes.io/version: 3.60.0
    helm.sh/chart: nexus-60.0.0
    jenkins.io/credentials-type: usernamePassword
  name: nexus
EOF
) | kubectl apply --namespace=$NAMESPACE -f -

Create the secret containing the Chartmuseum credentials:

(
cat << EOF
apiVersion: v1
kind: Secret
type: Opaque
data:
  BASIC_AUTH_PASS: ********
  BASIC_AUTH_USER: ********
metadata:
  annotations:
    meta.helm.sh/release-name: chartmuseum
    meta.helm.sh/release-namespace: $NAMESPACE
  labels:
    app.kubernetes.io/instance: chartmuseum
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: chartmuseum
    helm.sh/chart: chartmuseum-1.1.7
    jenkins.io/credentials-type: usernamePassword
  name: chartmuseum
EOF
) | kubectl apply --namespace=$NAMESPACE -f -

Create the secret containing the AWS credentials:

(
cat << EOF
apiVersion: v1
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: aws-credentials
    meta.helm.sh/release-namespace: $NAMESPACE
  labels:
    "app.kubernetes.io/managed-by": Helm
    aws-rotate-key: "true"
  name: aws-credentials
stringData:
  access_key_id: ********
  secret_access_key: ********
EOF
) | kubectl apply --namespace=$NAMESPACE -f -

The AWS credentials are rotated with the AWS IAM key rotate tool. The cron job schedule is configurable with the cronjob.schedule value.

Create the secret containing the Datadog API Key:

(
cat << EOF
apiVersion: v1
kind: Secret
type: Opaque
data:
  api-key: ********
metadata:
  annotations:
    meta.helm.sh/release-name: datadog
    meta.helm.sh/release-namespace: $NAMESPACE
  labels:
    app.kubernetes.io/managed-by: Helm
  name: datadog-nxio-api
EOF
) | kubectl apply --namespace=$NAMESPACE -f -

Misc

Create the ClusterRoleBinding required for the ServiceAccount used by Jenkins, typically to create namespaces:

(
cat << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: $NAMESPACE:jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: $NAMESPACE
EOF
) | kubectl apply -f -

Kubernetes Cluster Synchronization

The following environment variables need to be set:

• Credentials for nexus.platform.dev.nuxeo.com or nexus.platform-staging.dev.nuxeo.com:

  • NEXUS_USERNAME
  • NEXUS_PASSWORD

• Credentials for chartmuseum.platform.dev.nuxeo.com or chartmuseum.platform-staging.dev.nuxeo.com:

  • CHARTMUSEUM_USERNAME
  • CHARTMUSEUM_PASSWORD

• Credentials for packages.nuxeo.com:

  • PACKAGES_USERNAME
  • PACKAGES_PASSWORD

• Credentials for Nuxeo Connect:

  • CONNECT_USERNAME
  • CONNECT_PASSWORD

Synchronize the Kubernetes cluster with the resources from the helmfile.

helmfile deps
helmfile sync

The default Helmfile environment targets the platform-staging namespace. To target the platform namespace, specify the production environment:

helmfile deps
helmfile --environment production sync

See the environments section in the helmfile to understand the diff between the environments.

Have fun using Jenkins at https://jenkins.$NAMESPACE.dev.nuxeo.com/.