Merge pull request rook#13997 from BlaineEXE/upgrade-docs

doc: update release notes and upgrade docs for v1.14

Documentation/CRDs/Cluster/network-providers.md

@@ -155,7 +155,7 @@ understand and implement these requirements.
     need to be an order of magnitude larger (or more) than the host address space to allow the
     storage cluster to grow in the future.
 
-If these prerequisites are not achievable, the remaining option is to set the Rook operator config
+If these prerequisites are not achievable, plan to set the Rook operator config
 `CSI_ENABLE_HOST_NETWORK: "false"` as documented in the [CSI Host Networking](#csi-host-networking)
 section.
 
@@ -418,6 +418,24 @@ spec:
     }'
 ```
 
+## Holder Pod Deprecation
+
+Rook plans to remove CSI "holder" pods in Rook v1.16. CephCluster with `csi-*plugin-holder-*` pods
+present in the Rook operator namespace must plan to set `CSI_DISABLE_HOLDER_PODS` to `"true"` after
+Rook v1.14 is installed and before v1.16 is installed by following the migration sections below.
+CephClusters with no holder pods do not need to follow migration steps.
+
+CephClusters that do not use `network.provider: multus` can follow the
+[Disabling Holder Pods](#disabling-holder-pods) section.
+
+CephClusters that use `network.provider: multus` will need to plan the migration more carefully.
+Read the [Disabling Holder Pods with Multus and CSI Host Networking](#disabling-holder-pods-with-multus-and-csi-host-networking)
+section. Decide whether to use CSI host networking or not following the sections below.
+
+!!! hint
+    To determine if holder pods are deployed, use
+    `kubectl --namespace $ROOK_OPERATOR get pods | grep plugin-holder`
+
 ## Modifying CSI networking
 
 ### Disabling Holder Pods with Multus and CSI Host Networking
@@ -515,7 +533,8 @@ If any CephClusters have Multus enabled (`network.provider: "multus"`), follow t
 steps above before continuing.
 
 **Step 2**
-Begin by setting `CSI_DISABLE_HOLDER_PODS: "true"` (and `CSI_ENABLE_HOST_NETWORK: "true"` if desired).
+Begin by setting `CSI_DISABLE_HOLDER_PODS: "true"` -- also set the desired value of
+`CSI_ENABLE_HOST_NETWORK` if needed.
 
 After this, `csi-*plugin-*` pods will restart, and `csi-*plugin-holder-*` pods will remain running.
 

Documentation/Helm-Charts/operator-chart.md

@@ -81,7 +81,7 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.csiRBDPluginVolumeMount` | The volume mounts of the CephCSI RBD plugin DaemonSet | `nil` |
 | `csi.csiRBDProvisionerResource` | CEPH CSI RBD provisioner resource requirement list csi-omap-generator resources will be applied only if `enableOMAPGenerator` is set to `true` | see values.yaml |
 | `csi.disableCsiDriver` | Disable the CSI driver. | `"false"` |
-| `csi.disableHolderPods` | Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network without needing hosts to the network. Holder pods are being deprecated. See issue for details: https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true". | `true` |
+| `csi.disableHolderPods` | Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network without needing hosts to the network. Holder pods are being removed. See issue for details: https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true". | `true` |
 | `csi.enableCSIEncryption` | Enable Ceph CSI PVC encryption support | `false` |
 | `csi.enableCSIHostNetwork` | Enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary in some network configurations where the SDN does not provide access to an external cluster or there is significant drop in read/write performance | `true` |
 | `csi.enableCephfsDriver` | Enable Ceph CSI CephFS driver | `true` |

Documentation/Upgrade/rook-upgrade.md

@@ -14,14 +14,15 @@ We welcome feedback and opening issues!
 
 ## Supported Versions
 
-This guide is for upgrading from **Rook v1.12.x to Rook v1.13.x**.
+This guide is for upgrading from **Rook v1.13.x to Rook v1.14.x**.
 
 Please refer to the upgrade guides from previous releases for supported upgrade paths.
 Rook upgrades are only supported between official releases.
 
 For a guide to upgrade previous versions of Rook, please refer to the version of documentation for
 those releases.
 
+* [Upgrade 1.12 to 1.13](https://rook.io/docs/rook/v1.13/Upgrade/rook-upgrade/)
 * [Upgrade 1.11 to 1.12](https://rook.io/docs/rook/v1.12/Upgrade/rook-upgrade/)
 * [Upgrade 1.10 to 1.11](https://rook.io/docs/rook/v1.11/Upgrade/rook-upgrade/)
 * [Upgrade 1.9 to 1.10](https://rook.io/docs/rook/v1.10/Upgrade/rook-upgrade/)
@@ -47,22 +48,18 @@ those releases.
     official releases. Builds from the master branch can have functionality changed or removed at any
     time without compatibility support and without prior notice.
 
-## Breaking changes in v1.13
-
-* The minimum supported version of Kubernetes is v1.23.
-  Upgrade to Kubernetes v1.23 or higher before upgrading Rook.
-* The minimum supported version of Ceph is v17.2.0.
-  If a lower version is currently deployed, [Upgrade Ceph](./ceph-upgrade.md) before upgrading Rook.
-* CephCSI CephFS driver introduced a breaking change in v3.9.0. If any existing CephFS storageclass in
-  the cluster has `MountOptions` parameter set, follow the steps mentioned in the
-  [CephCSI upgrade guide](https://github.com/ceph/ceph-csi/blob/v3.9.0/docs/ceph-csi-upgrade.md/#upgrading-cephfs)
-  to ensure a smooth upgrade.
-  This became the default CSI version in Rook v1.12.1, and may have already been resolved.
-* Support for the admission controller has been removed. CRD validation is now enabled with
-  [Validating Admission Policies](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/).
-  Validating Admission Policy rules are ignored in Kubernetes v1.24 and lower.
-  If the admission controller is enabled, it is advised to upgrade to Kubernetes v1.25 or higher before upgrading Rook.
-  For more info, see https://github.com/rook/rook/pull/11532.
+## Breaking changes in v1.14
+
+* The minimum supported version of Kubernetes is v1.25.
+  Upgrade to Kubernetes v1.25 or higher before upgrading Rook.
+* The Rook operator config `CSI_ENABLE_READ_AFFINITY` was removed. v1.13 clusters that have modified
+  this value to be `"true"` must set the option as desired in each CephCluster as documented
+  [here](https://rook.github.io/docs/rook/v1.14/CRDs/Cluster/ceph-cluster-crd/#csi-driver-options)
+  before upgrading to v1.14.
+* Rook is beginning the process of deprecating CSI network "holder" pods.
+  If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, see the
+  [detailed documentation](../CRDs/Cluster/network-providers.md#holder-pod-deprecation)
+  to disable them. This is optional for v1.14, but will be required in a future release.
 
 ## Considerations
 
@@ -78,23 +75,23 @@ With this upgrade guide, there are a few notes to consider:
 
 Unless otherwise noted due to extenuating requirements, upgrades from one patch release of Rook to
 another are as simple as updating the common resources and the image of the Rook operator. For
-example, when Rook v1.13.1 is released, the process of updating from v1.13.0 is as simple as running
+example, when Rook v1.14.1 is released, the process of updating from v1.14.0 is as simple as running
 the following:
 
 ```console
-git clone --single-branch --depth=1 --branch v1.13.1 https://github.com/rook/rook.git
+git clone --single-branch --depth=1 --branch v1.14.1 https://github.com/rook/rook.git
 cd rook/deploy/examples
 ```
 
 If the Rook Operator or CephCluster are deployed into a different namespace than
 `rook-ceph`, see the [Update common resources and CRDs](#1-update-common-resources-and-crds)
 section for instructions on how to change the default namespaces in `common.yaml`.
 
-Then, apply the latest changes from v1.13, and update the Rook Operator image.
+Then, apply the latest changes from v1.14, and update the Rook Operator image.
 
 ```console
 kubectl apply -f common.yaml -f crds.yaml
-kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.13.1
+kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.14.1
 ```
 
 As exemplified above, it is a good practice to update Rook common resources from the example
@@ -109,6 +106,10 @@ Also update optional resources like Prometheus monitoring noted more fully in th
 If Rook is installed via the Helm chart, Helm will handle some details of the upgrade itself.
 The upgrade steps in this guide will clarify what Helm handles automatically.
 
+!!! important
+    If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, set the new
+    config `disableHolderPods: false` in the values.yaml before upgrading to v1.14.
+
 The `rook-ceph` helm chart upgrade performs the Rook upgrade.
 The `rook-ceph-cluster` helm chart upgrade performs a [Ceph upgrade](#ceph-version-upgrades) if the Ceph image is updated.
 The `rook-ceph` chart should be upgraded before `rook-ceph-cluster`, so the latest operator has the opportunity to update
@@ -128,9 +129,9 @@ In order to successfully upgrade a Rook cluster, the following prerequisites mus
 
 ## Rook Operator Upgrade
 
-The examples given in this guide upgrade a live Rook cluster running `v1.12.9` to
-the version `v1.13.0`. This upgrade should work from any official patch release of Rook v1.12 to any
-official patch release of v1.13.
+The examples given in this guide upgrade a live Rook cluster running `v1.13.7` to
+the version `v1.14.0`. This upgrade should work from any official patch release of Rook v1.13 to any
+official patch release of v1.14.
 
 Let's get started!
 
@@ -191,7 +192,7 @@ kubectl apply -f deploy/examples/monitoring/rbac.yaml
 !!! hint
     The operator is automatically updated when using Helm charts.
 
-The largest portion of the upgrade is triggered when the operator's image is updated to `v1.13.x`.
+The largest portion of the upgrade is triggered when the operator's image is updated to `v1.14.x`.
 When the operator is updated, it will proceed to update all of the Ceph daemons.
 
 ```console
@@ -225,18 +226,18 @@ watch --exec kubectl -n $ROOK_CLUSTER_NAMESPACE get deployments -l rook_cluster=
 ```
 
 As an example, this cluster is midway through updating the OSDs. When all deployments report `1/1/1`
-availability and `rook-version=v1.13.0`, the Ceph cluster's core components are fully updated.
+availability and `rook-version=v1.14.0`, the Ceph cluster's core components are fully updated.
 
 ```console
 Every 2.0s: kubectl -n rook-ceph get deployment -o j...
 
-rook-ceph-mgr-a         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-mon-a         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-mon-b         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-mon-c         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-osd-0         req/upd/avl: 1//        rook-version=v1.13.0
-rook-ceph-osd-1         req/upd/avl: 1/1/1      rook-version=v1.12.9
-rook-ceph-osd-2         req/upd/avl: 1/1/1      rook-version=v1.12.9
+rook-ceph-mgr-a         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-mon-a         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-mon-b         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-mon-c         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-osd-0         req/upd/avl: 1//        rook-version=v1.14.0
+rook-ceph-osd-1         req/upd/avl: 1/1/1      rook-version=v1.13.7
+rook-ceph-osd-2         req/upd/avl: 1/1/1      rook-version=v1.13.7
 ```
 
 An easy check to see if the upgrade is totally finished is to check that there is only one
@@ -245,23 +246,21 @@ An easy check to see if the upgrade is totally finished is to check that there i
 ```console
 # kubectl -n $ROOK_CLUSTER_NAMESPACE get deployment -l rook_cluster=$ROOK_CLUSTER_NAMESPACE -o jsonpath='{range .items[*]}{"rook-version="}{.metadata.labels.rook-version}{"\n"}{end}' | sort | uniq
 This cluster is not yet finished:
-  rook-version=v1.12.9
-  rook-version=v1.13.0
+  rook-version=v1.13.7
+  rook-version=v1.14.0
 This cluster is finished:
-  rook-version=v1.13.0
+  rook-version=v1.14.0
 ```
 
 ### **5. Verify the updated cluster**
 
-At this point, the Rook operator should be running version `rook/ceph:v1.13.0`.
+At this point, the Rook operator should be running version `rook/ceph:v1.14.0`.
 
 Verify the CephCluster health using the [health verification doc](health-verification.md).
 
-### **6. Disable CSI holder pods**
-CSI "holder" pods are frequently reported objects of confusion and struggle in Rook. Because of
-this, they are being deprecated and will be removed in Rook v1.16.
+### **6. Disable holder pods**
 
-If there are any CephClusters that use the non-default network setting `network.provider: "multus"`,
-or if the operator config `CSI_ENABLE_HOST_NETWORK` is set to `"false"`, perform migration steps to
-remove holder pods by setting `CSI_REMOVE_HOLDER_PODS: "true"` after following this migration guide:
-[Modifying CSI Networking](./../CRDs/Cluster/network-providers.md#modifying-csi-networking)
+Rook is beginning the process of deprecating CSI network "holder" pods. If there are pods named
+`csi-*plugin-holder-*` in the Rook operator namespace, see the
+[detailed documentation](../CRDs/Cluster/network-providers.md#holder-pod-deprecation)
+to disable them. This is optional for v1.14, but will be required in a future release.

PendingReleaseNotes.md

@@ -2,15 +2,16 @@
 
 ## Breaking Changes
 
-- The removal of `CSI_ENABLE_READ_AFFINITY` option and its replacement with per-cluster
-read affinity setting in cephCluster CR (CSIDriverOptions section) in [PR](https://github.com/rook/rook/pull/13665)
-- updating `netNamespaceFilePath` for all clusterIDs in rook-ceph-csi-config configMap in [PR](https://github.com/rook/rook/pull/13613)
-  - Issue: The netNamespaceFilePath isn't updated in the CSI config map for all the clusterIDs when `CSI_ENABLE_HOST_NETWORK` is set to false in `operator.yaml`
-  - Impact: This results in the unintended network configurations, with pods using the host networking instead of pod networking.
-- Rook is beginning the process of deprecating holder pods. This is especially important for Multus
-  users. Helm chart users should take care to set the new config `disableHolderPods: false` if they
-  are using Multus and still using the `rook-ceph` chart's default `values.yaml`. Special upgrade
-  docs for multus can be found [here](Documentation/CRDs/Cluster/network-providers.md#migrating-to-remove-multus-holder-pods).
+- The minimum supported version of Kubernetes is v1.25.
+  Upgrade to Kubernetes v1.25 or higher before upgrading Rook.
+- The Rook operator config `CSI_ENABLE_READ_AFFINITY` was removed. v1.13 clusters that have modified
+  this value to be `"true"` must set the option as desired in each CephCluster as documented
+  [here](https://rook.github.io/docs/rook/v1.14/CRDs/Cluster/ceph-cluster-crd/#csi-driver-options)
+  before upgrading to v1.14.
+- Rook is beginning the process of deprecating CSI network "holder" pods.
+  If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, see the
+  [detailed documentation](./Documentation/CRDs/Cluster/network-providers.md#holder-pod-deprecation)
+  to disable them. This is optional for v1.14, but will be required in a future release.
 
 ## Features
 

deploy/charts/rook-ceph/values.yaml

@@ -89,7 +89,7 @@ csi:
   # there is significant drop in read/write performance
   enableCSIHostNetwork: true
   # -- Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being deprecated. See issue for details:
+  # without needing hosts to the network. Holder pods are being removed. See issue for details:
   # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
   disableHolderPods: true
   # -- Enable Snapshotter in CephFS provisioner pod

deploy/examples/operator-openshift.yaml

@@ -140,7 +140,7 @@ data:
   # CSI_ENABLE_HOST_NETWORK: "true"
 
   # Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being deprecated. See issue for details:
+  # without needing hosts to the network. Holder pods are being removed. See issue for details:
   # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
   CSI_DISABLE_HOLDER_PODS: "true"
 

deploy/examples/operator.yaml

@@ -47,7 +47,7 @@ data:
   # CSI_ENABLE_HOST_NETWORK: "true"
 
   # Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being deprecated. See issue for details:
+  # without needing hosts to the network. Holder pods are being removed. See issue for details:
   # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
   CSI_DISABLE_HOLDER_PODS: "true"