Proposal: File/Directory/Patch based promotions

[jessesuen]

Proposed Feature

There are use cases of promotions where the act of promotion may include copying files or directories from one path to another. Additionally, or alternatively, the promotion may need to modify/patch files in a certain way as part of the promotion.

Conceptually, this is no different than what already happens when Kargo promotes container images on the main branch using kustomize, at different paths:

# promote to dev
$ cd env/dev && kustomize edit set image acme/guestbook:v2.0.0

# promote to staging
$ cd env/staging && kustomize edit set image acme/guestbook:v2.0.0

# promote to prod
$ cd env/prod && kustomize edit set image acme/guestbook:v2.0.0

For each promotion, you could describe what Kargo is doing as promoting a "patch" to kustomization.yaml files from path to path, where the patch is upserting an entry in the images: stanza:

images:
- name: acme/guestbook
  newTag: v2.0.0

This proposal is to generalize the ability to modify files into something that is not kustomize/kubernetes specific, and could apply to anything configuration-related:

• json
• yaml
• hcl

Motivation

Users of Kargo shouldn't be limited to, or beholden to the choice of tooling Kargo natively decided to support. This feature will expand the scope and capabilities of what Kargo is able to promote, such as Terraform.

Suggested Implementation

Largely TBD but I imagine we'll need ways to describe:

• filepaths
• exclusion filters
• extraction/parsing logic
• ways to manipulate a configuration file (e.g. yq, jq, sed)

Open Issues

When we move to support patch-based promotion, I think Kargo has to start treating "rollback" as something different than "promotion". Because while we may easily know how to apply a patch as part of the promotion, we would not easily know how to "undo" that patch for a rollback.

Ultimately, I think we may need to require the use of rendered branches to support rollback, and then rollback becomes nothing more than re-applying the contents of an older commit in that branch as the tip.

[jessesuen]

I forgot about one use case that we even practice internally. I would like a way for kargo to monitor git-repo-1/path for changes, and then whenever changes are made in that path, copy/replace the directory contents of git-repo-1/path to git-repo-2/path.

We constantly make changes/adjustments to helm chart contents in an upstream git repo. Let's say:
https://github.com/akuity/kargo/tree/main/charts/kargo

We infrequently release the helm chart (once every few weeks), but would still like a way to immediately deploy the changes of that helm chart before any release is made (whenever changes are made to chart/kargo directory in git).

You could think of this use case as similar to constantly building and pushing a container image at the tip of main, but instead of a container image, it's the contents of a file path in git.

[jmgilman]

Apologies if I am misunderstanding the proposal, but looking for a goldilocks solution seems infeasible given the wide range of activities that must be accounted for in the act of promotion. Wouldn't it be better to implement some contractual API akin to Argo's Custom Management Plugin interface? If I recall correctly, that was born out of trying to avoid people having to hack the repo server with all sorts of workarounds due to Argo's limited support for anything other than Helm/Kustomize.

We don't currently use Kargo (due to this exact limitation), but from my conceptual understanding, the "before" and "after" that the CMP primarily revolves around isn't much different here.

[krancour]

@jmgilman this and promotion mechanism plugins aren't mutually exclusive.

Plugins are definitely happening. The output of this issue could end up being a plugin.

[jmgilman]

Gotcha, thanks for clarifying. Is there an issue tracking plugins?

[krancour]

@jmgilman oddly there isn't. It's an epic anyway. But it is on the roadmap with plans for us to be designing during the current release cycle and implementing in the next.

[jessesuen]

Sharing one use case where patch promotion would be useful in the case of canarying.

A user could use a Stage to represent a traffic percentage of a production environment (e.g. 10% of traffic). For example, an Istio VirtualService could be predefined/predivided into 10% traffic split buckets. There would be 10 Stages, each representing a bucket it was responsible for controlling. A promotion could be defined as a YAML patch to modify the specific portion of that VirtualService, to ensure somehow that Stage's 10% were seeing the new version.

Similarly, a patch promotion could be useful in a feature gate scenario for a ringed deployment. Stages could be used to represent a portion of production users that would be exposed to a new feature (e.g. dogfood -> friendly users -> everyone). The promotion patch would set some configuration value to expose those users to the new feature.

[jessesuen]

I am capturing more use cases after some discussion with interested users at KubeCon. Based on some feedback, it is clear that the manner in which we patch files, needs to become more flexible.

Here are some use cases:

1.

A Freight might have been generated from a Warehouse subscribing to a helm chart. In this case semantic version would need to be updated in a file somewhere. Today, we support patching the semantic version of a dependency of an umbrella helm chart in git. I am arguing that the file we update should be arbitrary (e.g. it could be a kustomization.yaml where the semantic version is updated in helmCharts[0].version:

helmCharts:
- name: minecraft
  includeCRDs: false
  valuesInline:
    minecraftServer:
      eula: true
      difficulty: hard
      rcon:
        enabled: true
  releaseName: moria
  version: 3.1.3

2.

Another user explained, they need the Helm chart's semantic version to be updated in either an Application or ApplicationSet specification in git, since they Argo CD's feature to specify a chart + version.

Rather than our current native support for umbrella charts in git, I feel the kustomization.yaml update and Application/ApplicationSet update are equally as valid as the umbrella chart update. So we should create a generic mechanism such that all three can be patched.

[jessesuen]

For git repo updates, the promotion update mechanism spec should be able to reference the files in the git commit it wants to extract/copy, and then the destination files it wants to update/replace:

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: prod
spec:
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/example/kargo-demo.git
      patches:
        - sourceFiles:
           - base/values.yaml
           destinationFiles:
           - env/prod/values.yaml

In addition to replacing files/directories wholesale, we could extend this to extracting portions of the files in commit, then patching the destination files with expressions/tools like yq, jq, etc...

[vavdoshka]

Is Custom Management Plugin something considered to be release in 0.5.0 / 0.6.0 or this is something completely separate from this issue?

[krancour]

@vavdoshka we're still committed to pluggable promotion mechanisms, but we are re-prioritizing slightly in favor of stability of what we have so far. We are still having occasional discussions about it -- trying to agree internally on a path forward. fwiw, they probably aren't going to end up looking exactly like Argo CD CMPs.

[krancour]

After a lot of discussion, this feature makes the most sense after #1680 is completed.

[danielloader]

Now that is completed is the project in a position to be fishing for potential usecases to get some additional input? Given the feature is so open ended.

[krancour]

@danielloader we're hard at work on another critical pivot (#2219) and the design proposal we are currently working on internally includes file copy as on of the built-in directives.

[krancour]

@jessesuen do you want to weigh in on whether the copy promotion step covers our bases adequately enough to close this?

[jessesuen]

Let's leave this open until we get a better handle/feedback if copy can fulfill the use case. But until then, we don't have to schedule this and can leave this open without milestone.

[Chewie]

A use case where a generic patch via something equivalent to yq would be useful, and that copy doesn't cover, is a scenario in which the organization uses a CRD to represent the desired state (e.g OAM, knative, or a bespoke one created with crossplane), without helm or kustomize.

In this scenario, my gitops repo is just one document per stage, and I need to update the image reference in that CRD during the promotion step.

[danielloader]

A use case where a generic patch via something equivalent to yq would be useful, and that copy doesn't cover, is a scenario in which the organization uses a CRD to represent the desired state (e.g OAM, knative, or a bespoke one created with crossplane), without helm or kustomize.

In this scenario, my gitops repo is just one document per stage, and I need to update the image reference in that CRD during the promotion step.

Exactly this, for me I just want to update a YAML config that launched images are derived from - not too dissimilar to knative.

https://www.shinyproxy.io/documentation/configuration/#overview

.specs[].container-image in this scenario

[krancour]

@Chewie @danielloader I think what the two of you need is simpler than what @jessesuen was going for with this issue. You just need a promotion step that can perform arbitrary updates on a YAML document. That could be added with very little effort today, although its usefulness would be somewhat limited to changing an arbitrary key in an arbitrary YAML file to some value derived from your Freight. Once the EL support we're working on for this release is complete, this would be far more flexible. If there isn't already an issue open for adding such a step in conjunction with the EL, I will open one, as it would actually be a great way to show off what sort of things can be accomplished with the EL.

[jeffmccune]

@krancour

If there isn't already an issue open for adding such a step in conjunction with the EL, I will open one, as it would actually be a great way to show off what sort of things can be accomplished with the EL.

Could you help me find this issue? I'm looking into this to integrate Holos with Kargo as described in #2995

I found this from #527 which is also about integrating with CUE. Editing a yaml or json file with CEL would accomplish part of the integration, but I'm still looking for a way to run holos render platform as a pipeline step to fully render the manifests.

[krancour]

It's been fully addressed in #2941

[jmgilman]

So, just to be clear, we only have support for Kustomize, Helm, and plain YAML? Are we still missing tying into the rest of the ecosystem? All of my teams use intermediate tooling for generating the final YAML (CUE, KCL, etc.). It still looks like Kargo is incapable of being integrated into more complex deployments.

[krancour]

@jmgilman we do not plan on directly integrating with tool after tool. We plan on enabling development of custom/third-party promotion steps to allow these integrations to be developed outside of Kargo proper.

Discussion in this thread has gotten a bit sidetracked. Everyone, please let's keep this issue about really low-level, generic sort of things and please open other issues for things that aren't as low-level as copying a file or performing an arbitrary update on a YAML or JSON file.

[jmgilman]

@krancour I'm just really lost at where this feature is being tracked. I understand there's not going to be white-glove support for every tool out there, but there's been existing discussions around implementing plugins akin to Argo CD to be able to support these generators.

This issue briefly talked about it but was closed as a duplicate of the current issue. I also asked about this earlier in the thread but the answer really wasn't clear to me.

Is this plugin feature being tracked in this issue? Is there another one, yet? I'm just trying to have a place to track this plugin feature because our team would like to adopt Kargo when it can support our more advanced generation pipelines.

[jeffmccune]

@krancour

please let's keep this issue about really low-level, generic sort of things and please open other issues for things that aren't as low-level as copying a file or performing an arbitrary update on a YAML or JSON file.

To your point, I think @jmgilman and I need two things. First, perform an arbitrary update on a YAML or JSON file. The subject of this issue. Then, we need to execute a command other than what is built into Kargo to render manifests. Some other issue.

Is there an issue or discussion we can subscribe to and participate in regarding that second step, running the command after updating the YAML or JSON file? I opened up #2995 to try and find where the feature was being discussed as well.

[krancour]

It's been on the roadmap for some time, but did not have an issue. I just opened #2996.