Configuring Lagoon to deploy different environments to different target clusters

[tobybellwood]

With an increase in projects, it's become necessary to target different environments within a project to different Kubernetes clusters. In conversation with @shreddedbacon and @smlx here are our initial thoughts

Currently, the project defines a single existing kubernetes (openshift) cluster as a target for the environments to deploy onto.

This is then, in turn, accessed from environments by referencing the project that the environment is part of (GraphQL FTW)

What we are proposing is that instead of a project having a single target cluster, a project can instead have multiple destinations, defined by environmentType. i.e. if configured, development type environments could deploy into a different cluster than production type ones.

To achieve this, we would need to make the following changes:

• Allow projects to (optionally) configure a devKubernetes alongside a kubernetes. This would minimise impact for customers not using this new feature.
• On the initial deploy of a new environment, Lagoon would identify the target cluster from the project (usually kubernetes, but if the env is a development type, then devKubernetes)
• This target cluster ID would then be attached to the environment directly
• queries that currently access openshift IDs from environments via a project query would be rewritten to read the ID straight from the environment
• A migration script would be written to add the extra field into the environments that is initially populated from the current project configured kubernetes) meaning all environments are then correctly configured.
• Services that rely on access to the cluster (auto-idler, storage-calc, ssh, drush) etc would need to be backwards compatible.

Caveats:

• environments can't change clusters on-the-fly, they would have to be deleted and recreated on the new one
• UI/CLI tooling that currently assumes a single kubernetes for a project/env would need to consider this

Upsides:

• Projects wouldn't be lost between clusters in the event of a configuration change - an existing environment would always deploy where configured.
• Whilst we are initially defining the split based on production|development - the presence of the kubernetes ID on the environment object may open the path to richer configurability down the track, as essentially the process to define the ID only ever runs on initial deployment

Anything else we need to consider?

[Schnitzel]

We currently have two customers that already deploy into multiple kubernetes clusters from a single git repo. We solved this with creating two projects in lagoon. During this we realized that a simple production|development split of the projects is not enough.
As the customer didn't feel comfortable enough to just deploy the production environment to the production cluster we decided that we will also deploy a staging environment to the production cluster, so in the end it looks like this:

development k8s cluster:

• develop branch
• feature branches
• Pull Requests

production k8s cluster:

• master (this is their staging environment deployed from the master branch)
• production (this is promoted from the master environment)

So it would be great if we could figure out a way to handle such a system.

A couple of ideas I had:

1. we use the proposed production|development by Toby and just create a mutation via graphql that if you deploy an environment you can pass a kubernetes ID with it, IF the environment does not exist yet, it will be created on the provided kubernetes ID and ignoring the production|development split. IF the environment already exists we throw an error. - this solution would be "good enough for now"
2. We create a system where with regexes on the environment names you can define which environment is deployed into which cluster. This would probably be much more complex to setup, but also would be a system that allows more than just two kubernetes cluster to be supported.

Myself I would probably go with Nr 1 for now.

[tobybellwood]

Yes, 1 should not require too much extra consideration, and 2 could follow later in a rules based update that's backwards compatible.

[smlx]

1. will need to add some safety checks around the cluster that a project or environment is allowed to deploy to. I could see mistakes being made otherwise.

Regardless of the approach taken, this option adds significant complexity for users. We need to consider carefully how to expose this information in the UI and also document workflows that the system is designed to support.

[Schnitzel]

currently we handle it this way:
a user can see all clusters that have projects deployed into that the user has access too.
So basically as an lagoon admin I add a project to a cluster (as an admin I can see all clusters), which then subsequently the user can create a new project and assign the project to this cluster only.
I think for now this is the simplest and easiest way to do it?