From ce435ee7e3ecb316d0ff47f0aa7c8bd9fc136f39 Mon Sep 17 00:00:00 2001 From: "Benjamin A. Petersen" Date: Fri, 15 Sep 2023 17:19:58 -0400 Subject: [PATCH 1/5] Add proposal for Carvel Packages --- proposals/xxxx_carvel-package/README.md | 169 ++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 proposals/xxxx_carvel-package/README.md diff --git a/proposals/xxxx_carvel-package/README.md b/proposals/xxxx_carvel-package/README.md new file mode 100644 index 000000000..91d4747f5 --- /dev/null +++ b/proposals/xxxx_carvel-package/README.md @@ -0,0 +1,169 @@ +--- +title: "Carvel Package Management for Pinniped" +authors: [ "@benjaminapetersen" ] +status: "in-review" +sponsor: [ "@cfryanr", "@joshuatcasey" ] +approval_date: "" +--- + +*Disclaimer*: Proposals are point-in-time designs and decisions. +Once approved and implemented, they become historical documents. +If you are reading an old proposal, please be aware that the +features described herein might have continued to evolve since. + +# Carvel Package Management for Pinniped + +## Problem Statement + +There are a number of tools available to the Kubernetes ecosystem for deploying complex software +to a Kubernetes cluster. The Carvel toolchain provides a set of APIs, Custom Resources and CLI tools +that can aid a user in the configuration and lifecycle management of software deployed to a cluster. +We should enhance our deployment options by providing Carvel Packages that may be installed on a cluster +configured with `kapp-controller` to manage the software on the cluster. + +## How Pinniped Works Today (as of version v0.25.0) + +The `./deploy` directory in the root of the Pinniped repository contains a set of `ytt` templates +that: +- Are pre-rendered into installable templates listed with each Pinniped relese: + - [v0.25.0](https://github.com/vmware-tanzu/pinniped/releases/tag/v0.25.0) +- Can optionally be customized and rendered by a consumer of the Pinniped project by cloning down + the github repository, making changes to the `values.yaml` file and then rendered via `ytt`. + +## Terminology / Concepts + +- `Carvel` is an open-source project that provides tools for managing software build, configuration + and deployment on a Kubernetes cluster. For more information [read the Carvel docs](https://carvel.dev/). +- `kapp` is a Carvel provided CLI tool for deploying software onto a Kubernetes cluster. See + [the docs](https://carvel.dev/kapp/) for more information. +- `imgpkg` is a Carvel provided CLI tool that provides a mechanism for collecting configuration and + OCI images into a bundle that can be deployed on a cluster. +- `kapp-controller` is a server side component managing software in the form of Carvel `App`s delivered + via Carvel `Package`s. See [the docs](https://carvel.dev/kapp-controller/) for more information. +- `PackageRepository` is a custom resource that configures `kapp-controller` with a set of + `Package`s on a cluster. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package-repository) + for more information. +- `Package` is a custom resource that represents configuration in the form of metadata and OCI images + that may be used to deliver software onto a Kuberentes cluster. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package) + for more information. +- `PackageMetadata` is a custom resource describing attributes for a `Package`. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package-metadata) + for more information. + +## Proposal + +Allow Pinniped to be deployed onto a Kuberentes cluster through the mechanism of two Carvel `Packages`, +a Supervisor and a Concierge package. These may be delivered via a `PackageRepository` resource. + +### Goals and Non-goals + +Goals +- Provide an additional deployment option to deliver Pinniped software to a Kubernetes cluster + in the form of the `Package` apis provided by the Carvel toolchain. + +Non-Goals +- Provide additional deployment alternatives, such as official Helm charts + +#### API Changes + +No changes or additions to Pinniped APIs, this proposal represents a second, alternative +method for deployment utilising Carvel APIs. + +#### Upgrades + +Upgrading Pinniped via the Carvel Package mechanism will look something like this: + +- We deliver a `PackageRepository` that lists: + - Pinniped `Supervisor` package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) + - Pinniped `Concierge` package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) +- The user installs the Pinniped `PackageRepository`. +- The user creates: + - A `Supervisor` and `Concierge` `PackageInstall` Custom Resource (and `Secret`) with the following: +```yaml +spec: + packageRef: + refName: "supervisor.pinniped.dev" + versionSelection: + # - Constraints control the version and upgrades of the package + # if the constraint is pinned to a specific version, then only + # that version is installed on the cluster + # - If the PackageResource no longer serves a version that matches + # the constraint, then the PackageInstall will enter an error state + # until the constraint is updated + constraints: "0.25.0" + # - Alternatively, a constraint can be based on a semver range and can + # control automatic updates. + constraints: ">0.25.0" +``` + +#### Tests + +Our current integration test suite uses the `./deploy` directory to deploy Pinniped onto a +variety of clusters. We should continue to test this for the majority of users who do not +integrate with Carvel. In addition, we should update at least 1 of our tests to make use of +the new `Package` mechanism to ensure it functions correctly. + +Optionally we can: + +- Change a single test to deploy via the `Package` mechanism +- Change several tests to deploy via the `Package` mechanism +- Provide a flag to our `./prepare-for-integration-test.sh` and cycle all of our tests, + perhaps randomly back and forth between the simple deploy and the Package deploy. + +#### New Dependencies + +The Carvel toolset is not strickly a dependency for Pinniped itself. This proposal is an +optional method for delivering the Pinniped software to a kubernetes cluster. Therefore, `kapp`, +`kapp-controller`, and the custom resources such as `PackageRepository`, `Package`, `PackageMetadata`, +`PackageInstall` as well as `imgpkg`, `vendir` and `ytt` are all optional dependencies for a +consumer of Pinniped. + +#### Performance Considerations + +None. It is up to the user to determine if the adoption of the Carvel toolset is the +correct decision for their application lifecycle needs. + +#### Observability Considerations + +`Package`, `App` and `PackageInstall` custom resources contain a `status` field like many +Kubernetes resources. These are thoroughly detailed with relevant bits of information that may +aid the user in understanding the state of their applications. For example, a Carvel `App` Custom Resource +(disambiguation, a Carvel `app` (lowercase) from `kapp` vs an `App` (uppercase) from `kapp-controller` are +entirely different resources) contains a detailed status referencing all resources owned by the `App`. This +is very helpful when attempting to understand the state of a complex multi-component application. + +#### Security Considerations + +Carvel is a toolset separate from Pinniped. This feature is optional, users who choose to use +Carvel should assess Carvel for its risks and treadeoffs. + +#### Usability Considerations + +As of today the `./deploy` directory of Pinniped is implemented via the use of a subset of the +Carvel toolchain, namely, `ytt`. However, it is implemented in such a way that consumers of Pinniped +have choice, they may opt-in to the use of the Carvel toolchain, or simply `kubectl apply -f` our +pre-rendered yaml files, there is no requirement to use the Carvel toolchain. + +This feature serves the community users who have deeply adopted Carvel into their management, such that +`kapp-controller` is installed on their cluster and used to manage software lifecycle. + +#### Documentation Considerations + +This design doc serves as an announcement that the feature will be implemented. +It would be helpful to provide a blog post describing how the feature was validated. +Also include in release notes. + +#### Other Approaches Considered + +None. + +## Open Questions + +A list of questions that need to be answered. + +## Answered Questions + +## Implementation Plan + +## Implementation PRs + +* TBD From 4eb9a3d7ca52bd8f7e8e762a8b81a21aa3f64c6a Mon Sep 17 00:00:00 2001 From: "Benjamin A. Petersen" Date: Fri, 15 Sep 2023 17:39:14 -0400 Subject: [PATCH 2/5] Enhance proposal --- proposals/xxxx_carvel-package/README.md | 72 +++++++++++++++++++++++-- 1 file changed, 67 insertions(+), 5 deletions(-) diff --git a/proposals/xxxx_carvel-package/README.md b/proposals/xxxx_carvel-package/README.md index 91d4747f5..0043a6a70 100644 --- a/proposals/xxxx_carvel-package/README.md +++ b/proposals/xxxx_carvel-package/README.md @@ -18,8 +18,9 @@ features described herein might have continued to evolve since. There are a number of tools available to the Kubernetes ecosystem for deploying complex software to a Kubernetes cluster. The Carvel toolchain provides a set of APIs, Custom Resources and CLI tools that can aid a user in the configuration and lifecycle management of software deployed to a cluster. -We should enhance our deployment options by providing Carvel Packages that may be installed on a cluster -configured with `kapp-controller` to manage the software on the cluster. +We should enhance our deployment options by providing Carvel Packages for the `Suervisor` and `Concierge` +that may be installed on a cluster configured with Carvel's `kapp-controller` to manage the software +on the cluster. ## How Pinniped Works Today (as of version v0.25.0) @@ -30,6 +31,7 @@ that: - Can optionally be customized and rendered by a consumer of the Pinniped project by cloning down the github repository, making changes to the `values.yaml` file and then rendered via `ytt`. + ## Terminology / Concepts - `Carvel` is an open-source project that provides tools for managing software build, configuration @@ -52,21 +54,75 @@ that: ## Proposal Allow Pinniped to be deployed onto a Kuberentes cluster through the mechanism of two Carvel `Packages`, -a Supervisor and a Concierge package. These may be delivered via a `PackageRepository` resource. +a Supervisor and a Concierge package. These may be delivered via a `PackageRepository` resource and installed +via `PackageInstall` custom resources, and `Secret`s containing `Package` configuration. + +Conceptually, cluster managers would make the Pinniped software available on the +cluster by deploying the PackageRepository: + +```bash +# Deploy the Pinniped PackageRepository to the globally available +# namespace watched by kapp-controller for new Packages +kapp deploy --app pinniped-package-repository --file /pinniped-package-repository.yaml +``` + +Then developers responsible for deploying Supervisor and Concierge would create the +appropriate resources to successfully deploy the PackageInstall and Packages for both +`Supervisor` and `Concierge`: + +```bash +# create a Service account and RBAC for the PackageInstall +vim supervisor-service-and-rbac.yaml +kapp deploy --app supervisor-rbac --file supervisor-service-and-rbac.yaml +vim concierge-service-and-rbac.yaml +kapp deploy --app concierge-rbac --file concierge-service-and-rbac.yaml + +# create a PackageInstall and a Secret for configuring the Concierge +vim supervisor-package-install-bundle.yaml +kapp deploy --app supervisor --file supervisor-package-install-bundle.yaml +vim concierge-package-install-bundle.yaml +kapp deploy --app supervisor --file concierge-package-install-bundle.yaml +``` + +The `PackageRepository` will contain a series of versions of each of the Packages for Supervisor +and Concierge. + +The `PackageInstall` files will contain `constraints` representing acceptable versions of both the +`Supervisor` and `Concierge. For example: + +```yaml +spec: + packageRef: + # there will be two separate PackageInstall files, one for each + # Supervisor and Concierge + refName: "supervisor.pinniped.dev" + versionSelection: + # Constraints may be used to specify an exact version of the package + constraints: "0.25.0" + # Alternatively, a constraint can be based on a semver range and can + # specify multiple acceptible versions of the software. In this case, + # the Package will automatically upgrade to new versions when they become + # available, for example, when a new verison of the PackageRepository is + # deployed containing new versions of the Packages. + constraints: ">0.25.0" +``` ### Goals and Non-goals Goals - Provide an additional deployment option to deliver Pinniped software to a Kubernetes cluster in the form of the `Package` apis provided by the Carvel toolchain. +- Provide a `PackageRepository` and two separate `Package`s for Supervisor and Concierge. Non-Goals - Provide additional deployment alternatives, such as official Helm charts +- Provide a single package for both Supervisor and Concierge. +- Provide Packages for testing tools, such as `local-user-authenticator`. #### API Changes -No changes or additions to Pinniped APIs, this proposal represents a second, alternative -method for deployment utilising Carvel APIs. +No changes or additions to Pinniped's own APIs, this proposal represents a second, alternative +method for deployment utilising Carvel APIs and tools. #### Upgrades @@ -162,8 +218,14 @@ A list of questions that need to be answered. ## Answered Questions +* TBD - [Consult the open issue](https://github.com/vmware-tanzu/pinniped/issues/1614) requesting + the creation of this proposal + ## Implementation Plan +* TBD + ## Implementation PRs * TBD +* Consult the [Proof of concept WIP PR](https://github.com/vmware-tanzu/pinniped/pull/1635) From 39865452622e020aaab702c1b4fb6dd695e06714 Mon Sep 17 00:00:00 2001 From: "Benjamin A. Petersen" Date: Tue, 31 Oct 2023 11:55:47 -0400 Subject: [PATCH 3/5] typos --- proposals/xxxx_carvel-package/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/proposals/xxxx_carvel-package/README.md b/proposals/xxxx_carvel-package/README.md index 0043a6a70..857c6dc3e 100644 --- a/proposals/xxxx_carvel-package/README.md +++ b/proposals/xxxx_carvel-package/README.md @@ -18,7 +18,7 @@ features described herein might have continued to evolve since. There are a number of tools available to the Kubernetes ecosystem for deploying complex software to a Kubernetes cluster. The Carvel toolchain provides a set of APIs, Custom Resources and CLI tools that can aid a user in the configuration and lifecycle management of software deployed to a cluster. -We should enhance our deployment options by providing Carvel Packages for the `Suervisor` and `Concierge` +We should enhance our deployment options by providing Carvel Packages for the Supervisor and Concierge that may be installed on a cluster configured with Carvel's `kapp-controller` to manage the software on the cluster. @@ -26,7 +26,7 @@ on the cluster. The `./deploy` directory in the root of the Pinniped repository contains a set of `ytt` templates that: -- Are pre-rendered into installable templates listed with each Pinniped relese: +- Are pre-rendered into installable templates listed with each Pinniped release: - [v0.25.0](https://github.com/vmware-tanzu/pinniped/releases/tag/v0.25.0) - Can optionally be customized and rendered by a consumer of the Pinniped project by cloning down the github repository, making changes to the `values.yaml` file and then rendered via `ytt`. @@ -46,14 +46,14 @@ that: `Package`s on a cluster. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package-repository) for more information. - `Package` is a custom resource that represents configuration in the form of metadata and OCI images - that may be used to deliver software onto a Kuberentes cluster. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package) + that may be used to deliver software onto a Kubernetes cluster. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package) for more information. - `PackageMetadata` is a custom resource describing attributes for a `Package`. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package-metadata) for more information. ## Proposal -Allow Pinniped to be deployed onto a Kuberentes cluster through the mechanism of two Carvel `Packages`, +Allow Pinniped to be deployed onto a Kubernetes cluster through the mechanism of two Carvel `Packages`, a Supervisor and a Concierge package. These may be delivered via a `PackageRepository` resource and installed via `PackageInstall` custom resources, and `Secret`s containing `Package` configuration. @@ -68,7 +68,7 @@ kapp deploy --app pinniped-package-repository --file /pi Then developers responsible for deploying Supervisor and Concierge would create the appropriate resources to successfully deploy the PackageInstall and Packages for both -`Supervisor` and `Concierge`: +Supervisor and Concierge: ```bash # create a Service account and RBAC for the PackageInstall @@ -88,7 +88,7 @@ The `PackageRepository` will contain a series of versions of each of the Package and Concierge. The `PackageInstall` files will contain `constraints` representing acceptable versions of both the -`Supervisor` and `Concierge. For example: +Supervisor and Concierge. For example: ```yaml spec: @@ -129,11 +129,11 @@ method for deployment utilising Carvel APIs and tools. Upgrading Pinniped via the Carvel Package mechanism will look something like this: - We deliver a `PackageRepository` that lists: - - Pinniped `Supervisor` package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) - - Pinniped `Concierge` package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) + - Pinniped Supervisor package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) + - Pinniped Concierge package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) - The user installs the Pinniped `PackageRepository`. - The user creates: - - A `Supervisor` and `Concierge` `PackageInstall` Custom Resource (and `Secret`) with the following: + - A Supervisor and Concierge `PackageInstall` Custom Resource (and `Secret`) with the following: ```yaml spec: packageRef: From aa32f0a8b742da297629a6b99fd6066708373114 Mon Sep 17 00:00:00 2001 From: "Benjamin A. Petersen" Date: Tue, 31 Oct 2023 12:01:24 -0400 Subject: [PATCH 4/5] typos --- proposals/xxxx_carvel-package/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/xxxx_carvel-package/README.md b/proposals/xxxx_carvel-package/README.md index 857c6dc3e..35ab1a847 100644 --- a/proposals/xxxx_carvel-package/README.md +++ b/proposals/xxxx_carvel-package/README.md @@ -100,9 +100,9 @@ spec: # Constraints may be used to specify an exact version of the package constraints: "0.25.0" # Alternatively, a constraint can be based on a semver range and can - # specify multiple acceptible versions of the software. In this case, + # specify multiple acceptable versions of the software. In this case, # the Package will automatically upgrade to new versions when they become - # available, for example, when a new verison of the PackageRepository is + # available, for example, when a new version of the PackageRepository is # deployed containing new versions of the Packages. constraints: ">0.25.0" ``` @@ -167,7 +167,7 @@ Optionally we can: #### New Dependencies -The Carvel toolset is not strickly a dependency for Pinniped itself. This proposal is an +The Carvel toolset is not strictly a dependency for Pinniped itself. This proposal is an optional method for delivering the Pinniped software to a kubernetes cluster. Therefore, `kapp`, `kapp-controller`, and the custom resources such as `PackageRepository`, `Package`, `PackageMetadata`, `PackageInstall` as well as `imgpkg`, `vendir` and `ytt` are all optional dependencies for a @@ -190,13 +190,13 @@ is very helpful when attempting to understand the state of a complex multi-compo #### Security Considerations Carvel is a toolset separate from Pinniped. This feature is optional, users who choose to use -Carvel should assess Carvel for its risks and treadeoffs. +Carvel should assess Carvel for its risks and tradeoffs. #### Usability Considerations As of today the `./deploy` directory of Pinniped is implemented via the use of a subset of the Carvel toolchain, namely, `ytt`. However, it is implemented in such a way that consumers of Pinniped -have choice, they may opt-in to the use of the Carvel toolchain, or simply `kubectl apply -f` our +have choice, they may opt in to the use of the Carvel toolchain, or simply `kubectl apply -f` our pre-rendered yaml files, there is no requirement to use the Carvel toolchain. This feature serves the community users who have deeply adopted Carvel into their management, such that From fdc0eef16ca7264e7d046ca211f21f85be597be4 Mon Sep 17 00:00:00 2001 From: "Benjamin A. Petersen" Date: Tue, 31 Oct 2023 12:08:54 -0400 Subject: [PATCH 5/5] revised wording --- proposals/xxxx_carvel-package/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/xxxx_carvel-package/README.md b/proposals/xxxx_carvel-package/README.md index 35ab1a847..236719bbc 100644 --- a/proposals/xxxx_carvel-package/README.md +++ b/proposals/xxxx_carvel-package/README.md @@ -29,12 +29,12 @@ that: - Are pre-rendered into installable templates listed with each Pinniped release: - [v0.25.0](https://github.com/vmware-tanzu/pinniped/releases/tag/v0.25.0) - Can optionally be customized and rendered by a consumer of the Pinniped project by cloning down - the github repository, making changes to the `values.yaml` file and then rendered via `ytt`. + the Github repository, making changes to the `values.yaml` file and then rendering via `ytt`. ## Terminology / Concepts -- `Carvel` is an open-source project that provides tools for managing software build, configuration +- `Carvel` is an set of open-source projects that provide tools for managing software build, configuration and deployment on a Kubernetes cluster. For more information [read the Carvel docs](https://carvel.dev/). - `kapp` is a Carvel provided CLI tool for deploying software onto a Kubernetes cluster. See [the docs](https://carvel.dev/kapp/) for more information.