Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decision: OCP minor version tracking #327

Merged
merged 5 commits into from
May 13, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
= OpenShift Minor Version Upgrade Tracking

== Problem

Upgrading OpenShift minor versions requires a lot of coordination between our OpenShift team, customers, and other teams like AppCat and AppFlow teams.
It's difficult to track the status of each cluster.
We need to weekly check if and when we can switch which cluster to a new version and not forget the required pull requests.
bastjan marked this conversation as resolved.
Show resolved Hide resolved

=== Goals

* Allowing an overview of the status of each cluster
* Automating the process of changing the release channel

=== Non-Goals

* Automating customer or team approval for the upgrade

== Proposals

=== Automation in Jira to set cluster facts or update the tenant git repository
bastjan marked this conversation as resolved.
Show resolved Hide resolved

Jira is the main tool for tracking customer communication and the status of the upgrade.

We link the Jira ticket to the cluster and set the Lieutenant cluster fact or update the tenant git repository with the required information.
bastjan marked this conversation as resolved.
Show resolved Hide resolved

We don't have much experience with Jira automation or workflow, so we don't know how much effort this would be.
Additionally Jira isn't in our operational control, so we would need to work with other equally busy teams to get this done.

We're also not if Jira is here to stay or if we will switch to another tool in the nearish future.
bastjan marked this conversation as resolved.
Show resolved Hide resolved

=== Automation in GitLab with metrics export to Grafana dashboard

We create automation to schedule a pull request merge in the tenant git repository to change the release channel.
bastjan marked this conversation as resolved.
Show resolved Hide resolved

The pull request automation would ideally also export metrics about the pull request status and expected merge time to a Grafana dashboard.

=== Automation in upgrade controller with Grafana dashboard

Allow the upgrade controller to schedule `ClusterVersion` changes.
This could be in the form of a base version with overlaid patches for the `ClusterVersion` resource.

We would need an additional metric for current channel and future versions.

[source,yaml]
----
apiVersion: managedupgrade.appuio.io/v1beta1
kind: ClusterVersion
metadata:
name: version
namespace: appuio-openshift-upgrade-controller
spec:
template:
spec:
channel: stable-4.14
clusterID: XXX
patches:
- from: "2024-07-12T00:00:00Z"
patch:
spec:
channel: stable-4.15
simu marked this conversation as resolved.
Show resolved Hide resolved
----

== Decision

We add automation to the upgrade controller to schedule `ClusterVersion` changes and implement the required metrics for the Grafana dashboard.

== Rationale

The upgrade controller is already in place and we've got experience with the codebase.

All relevant timestamps are in a single place and thus easier to track.

We don't want to build Jira automation or GitLab automation for a single use case.

== References

- https://github.com/appuio/openshift-upgrade-controller/blob/master/api/v1beta1/clusterversion_types.go
- xref:oc4:ROOT:explanations/decisions/scheduled-mr-merges.adoc[]
- https://ticket.vshn.net/browse/SYN-1387[Build automation for merging MRs at specified times (internal)]
1 change: 1 addition & 0 deletions docs/modules/ROOT/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@
*** xref:oc4:ROOT:explanations/decisions/multi-team-alert-routing-base-alerts.adoc[]
** xref:oc4:ROOT:explanations/decisions/shipping-metrics-to-centralized-instance.adoc[]
** xref:oc4:ROOT:explanations/decisions/scheduled-mr-merges.adoc[]
** xref:oc4:ROOT:explanations/decisions/ocp-minor-version-tracking.adoc[]
** xref:oc4:ROOT:explanations/decisions/subscription-tracking.adoc[]
** xref:oc4:ROOT:explanations/decisions/admin-kubeconfig.adoc[]
** xref:oc4:ROOT:explanations/decisions/cloudscale-cilium-egressip.adoc[]