Carvel Package Proposal #1678
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,231 @@ | ||
| --- | ||
| title: "Carvel Package Management for Pinniped" | ||
| authors: [ "@benjaminapetersen" ] | ||
| status: "in-review" | ||
| sponsor: [ "@cfryanr", "@joshuatcasey" ] | ||
| approval_date: "" | ||
| --- | ||
|
|
||
| *Disclaimer*: Proposals are point-in-time designs and decisions. | ||
| Once approved and implemented, they become historical documents. | ||
| If you are reading an old proposal, please be aware that the | ||
| features described herein might have continued to evolve since. | ||
|
|
||
| # Carvel Package Management for Pinniped | ||
|
|
||
| ## Problem Statement | ||
|
|
||
| There are a number of tools available to the Kubernetes ecosystem for deploying complex software | ||
| to a Kubernetes cluster. The Carvel toolchain provides a set of APIs, Custom Resources and CLI tools | ||
| that can aid a user in the configuration and lifecycle management of software deployed to a cluster. | ||
| We should enhance our deployment options by providing Carvel Packages for the Supervisor and Concierge | ||
| that may be installed on a cluster configured with Carvel's `kapp-controller` to manage the software | ||
| on the cluster. | ||
|
|
||
| ## How Pinniped Works Today (as of version v0.25.0) | ||
|
|
||
| The `./deploy` directory in the root of the Pinniped repository contains a set of `ytt` templates | ||
| that: | ||
| - Are pre-rendered into installable templates listed with each Pinniped release: | ||
| - [v0.25.0](https://github.com/vmware-tanzu/pinniped/releases/tag/v0.25.0) | ||
| - Can optionally be customized and rendered by a consumer of the Pinniped project by cloning down | ||
| the Github repository, making changes to the `values.yaml` file and then rendering via `ytt`. | ||
|
|
||
|
|
||
| ## Terminology / Concepts | ||
|
|
||
| - `Carvel` is an set of open-source projects that provide tools for managing software build, configuration | ||
| and deployment on a Kubernetes cluster. For more information [read the Carvel docs](https://carvel.dev/). | ||
| - `kapp` is a Carvel provided CLI tool for deploying software onto a Kubernetes cluster. See | ||
| [the docs](https://carvel.dev/kapp/) for more information. | ||
| - `imgpkg` is a Carvel provided CLI tool that provides a mechanism for collecting configuration and | ||
| OCI images into a bundle that can be deployed on a cluster. | ||
| - `kapp-controller` is a server side component managing software in the form of Carvel `App`s delivered | ||
| via Carvel `Package`s. See [the docs](https://carvel.dev/kapp-controller/) for more information. | ||
| - `PackageRepository` is a custom resource that configures `kapp-controller` with a set of | ||
| `Package`s on a cluster. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package-repository) | ||
| for more information. | ||
| - `Package` is a custom resource that represents configuration in the form of metadata and OCI images | ||
| that may be used to deliver software onto a Kubernetes cluster. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package) | ||
| for more information. | ||
| - `PackageMetadata` is a custom resource describing attributes for a `Package`. See [the docs](https://carvel.dev/kapp-controller/docs/v0.47.x/packaging/#package-metadata) | ||
|
There was a problem hiding this comment. This looks like the way to specify your custom There was a problem hiding this comment. As a consumer, to install the software, you'll use a ---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
name: "supervisor-package-install"
namespace: supervisor-install-namespace # should NOT be "supervisor", keep install stuff separate
spec:
serviceAccountName: "pinniped-package-rbac-supervisor" # will need RBAC
packageRef:
refName: "supervisor.pinniped.dev"
versionSelection:
constraints: ">0.25.0" # packages will auto upgrade if they can, up to this constraint
values:
- secretRef:
name: "supervisor-package-install-secret" # secret contains values.yaml values
---
apiVersion: v1
kind: Secret
metadata:
name: "supervisor-package-install-secret"
namespace: supervisor-install-namespace # should NOT be "supervisor", keep install stuff separate
# this looks familiar to us!
# this is where the actual values will live.
stringData:
values.yml: |
---
app_name: pinniped-supervisor
namespace: supervisor
api_group_suffix: pinniped.dev
image_repo: pinniped.local/test/build
image_tag: 6114CB3E-ACB5-4C67-ABE7-9ACC09B73056
log_level: debug
service_https_nodeport_port: 443
service_https_nodeport_nodeport: 31243
service_https_clusterip_port: 443
There was a problem hiding this comment. could we add this to the example There was a problem hiding this comment. Yup, I can expand on this. |
||
| for more information. | ||
|
|
||
| ## Proposal | ||
|
|
||
| Allow Pinniped to be deployed onto a Kubernetes cluster through the mechanism of two Carvel `Packages`, | ||
| a Supervisor and a Concierge package. These may be delivered via a `PackageRepository` resource and installed | ||
| via `PackageInstall` custom resources, and `Secret`s containing `Package` configuration. | ||
|
|
||
| Conceptually, cluster managers would make the Pinniped software available on the | ||
| cluster by deploying the PackageRepository: | ||
|
|
||
| ```bash | ||
| # Deploy the Pinniped PackageRepository to the globally available | ||
| # namespace watched by kapp-controller for new Packages | ||
| kapp deploy --app pinniped-package-repository --file <pinniped-release-files>/pinniped-package-repository.yaml | ||
| ``` | ||
|
|
||
| Then developers responsible for deploying Supervisor and Concierge would create the | ||
| appropriate resources to successfully deploy the PackageInstall and Packages for both | ||
| Supervisor and Concierge: | ||
|
|
||
| ```bash | ||
| # create a Service account and RBAC for the PackageInstall | ||
| vim supervisor-service-and-rbac.yaml | ||
| kapp deploy --app supervisor-rbac --file supervisor-service-and-rbac.yaml | ||
| vim concierge-service-and-rbac.yaml | ||
| kapp deploy --app concierge-rbac --file concierge-service-and-rbac.yaml | ||
|
|
||
| # create a PackageInstall and a Secret for configuring the Concierge | ||
| vim supervisor-package-install-bundle.yaml | ||
| kapp deploy --app supervisor --file supervisor-package-install-bundle.yaml | ||
| vim concierge-package-install-bundle.yaml | ||
| kapp deploy --app supervisor --file concierge-package-install-bundle.yaml | ||
| ``` | ||
|
|
||
| The `PackageRepository` will contain a series of versions of each of the Packages for Supervisor | ||
| and Concierge. | ||
|
|
||
| The `PackageInstall` files will contain `constraints` representing acceptable versions of both the | ||
| Supervisor and Concierge. For example: | ||
|
|
||
| ```yaml | ||
| spec: | ||
|
There was a problem hiding this comment. Would you mind including |
||
| packageRef: | ||
| # there will be two separate PackageInstall files, one for each | ||
| # Supervisor and Concierge | ||
| refName: "supervisor.pinniped.dev" | ||
| versionSelection: | ||
| # Constraints may be used to specify an exact version of the package | ||
| constraints: "0.25.0" | ||
| # Alternatively, a constraint can be based on a semver range and can | ||
| # specify multiple acceptable versions of the software. In this case, | ||
| # the Package will automatically upgrade to new versions when they become | ||
| # available, for example, when a new version of the PackageRepository is | ||
| # deployed containing new versions of the Packages. | ||
| constraints: ">0.25.0" | ||
| ``` | ||
|
|
||
| ### Goals and Non-goals | ||
|
|
||
| Goals | ||
|
There was a problem hiding this comment. At this point, should we update this section to indicate that this is not a fully supported feature and is instead meant to provide users both an example and a mechanism they may use to create their own package? |
||
| - Provide an additional deployment option to deliver Pinniped software to a Kubernetes cluster | ||
| in the form of the `Package` apis provided by the Carvel toolchain. | ||
| - Provide a `PackageRepository` and two separate `Package`s for Supervisor and Concierge. | ||
|
|
||
| Non-Goals | ||
| - Provide additional deployment alternatives, such as official Helm charts | ||
| - Provide a single package for both Supervisor and Concierge. | ||
| - Provide Packages for testing tools, such as `local-user-authenticator`. | ||
|
|
||
| #### API Changes | ||
|
|
||
| No changes or additions to Pinniped's own APIs, this proposal represents a second, alternative | ||
| method for deployment utilising Carvel APIs and tools. | ||
|
|
||
| #### Upgrades | ||
|
There was a problem hiding this comment. Could you include a note here about upgrading from a non-Carvel installation to a Carvel installation? There was a problem hiding this comment. I can. I think it would simply be backup your specific configuration, delete Pinniped & reinstall. |
||
|
|
||
| Upgrading Pinniped via the Carvel Package mechanism will look something like this: | ||
|
|
||
| - We deliver a `PackageRepository` that lists: | ||
| - Pinniped Supervisor package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) | ||
| - Pinniped Concierge package at a number of versions (ex: 0.23.0,0.24.0,0.25.0, etc) | ||
| - The user installs the Pinniped `PackageRepository`. | ||
| - The user creates: | ||
| - A Supervisor and Concierge `PackageInstall` Custom Resource (and `Secret`) with the following: | ||
| ```yaml | ||
| spec: | ||
| packageRef: | ||
| refName: "supervisor.pinniped.dev" | ||
| versionSelection: | ||
| # - Constraints control the version and upgrades of the package | ||
| # if the constraint is pinned to a specific version, then only | ||
| # that version is installed on the cluster | ||
| # - If the PackageResource no longer serves a version that matches | ||
| # the constraint, then the PackageInstall will enter an error state | ||
| # until the constraint is updated | ||
| constraints: "0.25.0" | ||
| # - Alternatively, a constraint can be based on a semver range and can | ||
| # control automatic updates. | ||
| constraints: ">0.25.0" | ||
| ``` | ||
|
|
||
| #### Tests | ||
|
|
||
| Our current integration test suite uses the `./deploy` directory to deploy Pinniped onto a | ||
| variety of clusters. We should continue to test this for the majority of users who do not | ||
| integrate with Carvel. In addition, we should update at least 1 of our tests to make use of | ||
| the new `Package` mechanism to ensure it functions correctly. | ||
|
|
||
| Optionally we can: | ||
|
|
||
| - Change a single test to deploy via the `Package` mechanism | ||
| - Change several tests to deploy via the `Package` mechanism | ||
| - Provide a flag to our `./prepare-for-integration-test.sh` and cycle all of our tests, | ||
| perhaps randomly back and forth between the simple deploy and the Package deploy. | ||
|
|
||
| #### New Dependencies | ||
|
|
||
| The Carvel toolset is not strictly a dependency for Pinniped itself. This proposal is an | ||
| optional method for delivering the Pinniped software to a kubernetes cluster. Therefore, `kapp`, | ||
| `kapp-controller`, and the custom resources such as `PackageRepository`, `Package`, `PackageMetadata`, | ||
| `PackageInstall` as well as `imgpkg`, `vendir` and `ytt` are all optional dependencies for a | ||
| consumer of Pinniped. | ||
|
|
||
| #### Performance Considerations | ||
|
|
||
| None. It is up to the user to determine if the adoption of the Carvel toolset is the | ||
| correct decision for their application lifecycle needs. | ||
|
|
||
| #### Observability Considerations | ||
|
|
||
| `Package`, `App` and `PackageInstall` custom resources contain a `status` field like many | ||
| Kubernetes resources. These are thoroughly detailed with relevant bits of information that may | ||
| aid the user in understanding the state of their applications. For example, a Carvel `App` Custom Resource | ||
| (disambiguation, a Carvel `app` (lowercase) from `kapp` vs an `App` (uppercase) from `kapp-controller` are | ||
| entirely different resources) contains a detailed status referencing all resources owned by the `App`. This | ||
| is very helpful when attempting to understand the state of a complex multi-component application. | ||
|
|
||
| #### Security Considerations | ||
|
|
||
| Carvel is a toolset separate from Pinniped. This feature is optional, users who choose to use | ||
| Carvel should assess Carvel for its risks and tradeoffs. | ||
|
|
||
| #### Usability Considerations | ||
|
|
||
| As of today the `./deploy` directory of Pinniped is implemented via the use of a subset of the | ||
| Carvel toolchain, namely, `ytt`. However, it is implemented in such a way that consumers of Pinniped | ||
| have choice, they may opt in to the use of the Carvel toolchain, or simply `kubectl apply -f` our | ||
| pre-rendered yaml files, there is no requirement to use the Carvel toolchain. | ||
|
|
||
| This feature serves the community users who have deeply adopted Carvel into their management, such that | ||
| `kapp-controller` is installed on their cluster and used to manage software lifecycle. | ||
|
|
||
| #### Documentation Considerations | ||
|
|
||
| This design doc serves as an announcement that the feature will be implemented. | ||
| It would be helpful to provide a blog post describing how the feature was validated. | ||
| Also include in release notes. | ||
|
|
||
| #### Other Approaches Considered | ||
|
|
||
| None. | ||
|
|
||
| ## Open Questions | ||
|
|
||
| A list of questions that need to be answered. | ||
|
|
||
| ## Answered Questions | ||
|
|
||
| * TBD - [Consult the open issue](https://github.com/vmware-tanzu/pinniped/issues/1614) requesting | ||
| the creation of this proposal | ||
|
|
||
| ## Implementation Plan | ||
|
|
||
| * TBD | ||
|
|
||
| ## Implementation PRs | ||
|
|
||
| * TBD | ||
| * Consult the [Proof of concept WIP PR](https://github.com/vmware-tanzu/pinniped/pull/1635) | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we document which resources will not be deployed by the carvel package? Some possibilities:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that sounds good.