`helmCharts` repo field doesn't work with OCI registry #4381

QuinnBast · 2022-01-11T17:12:20Z

Describe the bug

Kustomize fails to pull charts through helmCharts yaml that are hosted within an OCI registry.

** Example **

Create a kustomize.yaml file:

helmCharts:
  - name: someName
    includeCRDs: true
    valuesFile: values.yaml
    releaseName: myRelease
    version: 0.0.26
    repo: oci://myRegistry

Run the command kustomize build --enable-helm --helm-command $helmCommand ./folder

Kustomize will fail to pull the helm chart and produce the following error:

Error: Error: looks like "oci://myRegistry" is not a valid chart repository or cannot be reached: object required
: unable to run: '/usr/sbin/helm pull --untar --untardir <someDir> --repo oci://myRegistry myRelease --version 0.0.26' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-991395685/helm HELM_CACHE_HOME=/tmp/kustomize-helm-991395685/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-991395685/helm/.data] (is '/usr/sbin/helm' installed?)

Running a regular helm pull works fine on the registry, but pulling with --repo causes issues:

# Working
$ helm pull oci://myRegistry/myRelease --version 0.0.26
Pulled: myRegistry/myRelease:0.0.26
Digest: sha256:14e1bc1ec1d0147eb8b41f081faae2ecea63e862ad4259d3a9cd1c7d1584be63

# Not working with `--repo`
$ helm pull myRelease --version 0.0.26 --repo myRegistry
Error: could not find protocol handler for: 

$ helm pull myRelease --version 0.0.26 --repo oci://myRegistry
Error: looks like "oci://myRegistry" is not a valid chart repository or cannot be reached: object required

I also cannot work around this by just adding it as a repo:

$ helm repo add reg myRegistry
Error: could not find protocol handler for: 

$ helm repo add reg oci://myRegistry
Error: looks like "oci://myRegistry" is not a valid chart repository or cannot be reached: object required

Expected output

Expected Kustomize to be able to pull the chart and apply the values.yaml to the chart. The solution would be for oci:// urls, to not use the --repo flag when performing a helm pull.

Kustomize version

$ kustomize version
{Version:kustomize/v4.3.0 GitCommit:cd17338759ef64c14307991fd25d52259697f1fb BuildDate:2021-08-24T19:24:28Z GoOs:linux GoArch:amd64}

$ helm version
version.BuildInfo{Version:"v3.7.2", GitCommit:"663a896f4a815053445eec4153677ddc24a0a361", GitTreeState:"clean", GoVersion:"go1.16.10"}

Platform

Ubuntu

The text was updated successfully, but these errors were encountered:

KlavsKlavsen · 2022-01-27T07:34:40Z

Helm 3.8.0 was just released with the new OCI support moved to 'production' status - this implements the changes that was done in 3.7.1 as stable (no longer experiemental env needed to use it in Helm) :)

QuinnBast · 2022-02-09T23:30:54Z

I upgraded to helm 3.8.0 and am getting the same error. The use of --repo is problematic for oci registries.

$ helm version
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

$ kustomize build --enable-helm ./test
Error: Error: looks like "oci://myRegistry" is not a valid chart repository or cannot be reached: object required
: unable to run: 'helm pull --untar --untardir <someDir> --repo oci://myRegistry myRelease --version 0.0.26' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-991395685/helm HELM_CACHE_HOME=/tmp/kustomize-helm-991395685/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-991395685/helm/.data] (is 'helm' installed?)

Can pull without specifying --repo:

$ helm pull oci://myRegistry --version 0.0.26
Pulled: myRegistry/myRelease:0.0.26
Digest: sha256:14e1bc1ec1d0147eb8b41f081faae2ecea63e862ad4259d3a9cd1c7d1584be63

Cannot pull with --repo

$ helm pull myRelease --version 0.0.26 --repo myRegistry
Error: could not find protocol handler for: 

$ helm pull myRelease --version 0.0.26 --repo oci://myRegistry
Error: looks like "oci://myRegistry" is not a valid chart repository or cannot be reached: object required

This seems to be the issue. Maybe if the repo is oci:// you can make this not use the --repo flag?

QuinnBast · 2022-03-22T16:03:29Z

I am unable to make a MR because I can't sign the CLA but can someone make this MR for me?

This line should be:

func (p *HelmChartInflationGeneratorPlugin) pullCommand() []string {
	args := []string{
		"pull",
		"--untar",
		"--untardir", p.absChartHome(),
-               "--repo", p.Repo
-		p.Name}
+             }
	if p.Version != "" {
		args = append(args, "--version", p.Version)
	}
+	if strings.HasPrefix(p.Repo, "oci://") {
+	        args = append(args, p.Repo)
+	} else {
+              args = append(args, "--repo", p.Repo)
+              args = append(args, p.Name)
+       }
	return args
}

This is untested, but --repo should not be in the pull command if oci:// is specified. If oci:// is specified, just do a straight helm pull

tis-rpage · 2022-03-28T17:35:27Z

If anyone takes the above patch, I think they should note the trailing comma needs fixing:

- 		"--untardir", p.absChartHome(),
+ 		"--untardir", p.absChartHome()

I am unable to make a MR because I can't sign the CLA but can someone make this MR for me?

This line should be:

func (p *HelmChartInflationGeneratorPlugin) pullCommand() []string {
	args := []string{
		"pull",
		"--untar",
		"--untardir", p.absChartHome(),
-               "--repo", p.Repo
-		p.Name}
+             }
	if p.Version != "" {
		args = append(args, "--version", p.Version)
	}
+	if strings.HasPrefix(p.Repo, "oci://") {
+	        args = append(args, p.Repo)
+	} else {
+              args = append(args, "--repo", p.Repo)
+              args = append(args, p.Name)
+       }
	return args
}

This is untested, but --repo should not be in the pull command if oci:// is specified. If oci:// is specified, just do a straight helm pull

These fixes update the HelmChartInflation tool so that it works with Helm 3 These changes are related to issues kubernetes-sigs#4381 and kubernetes-sigs#4335 on the kustomize github

minnie-jeong-otsk · 2022-04-05T07:23:54Z

It'd be greatly appreciated if this fix is deployed soon!

tis-rpage · 2022-04-05T13:27:18Z

To be fair, this work is required because the Helm team doesn't follow SemVer with their tooling. They made a minor release with major CLI interface changes, which kustomize should not be responsible for fixing.
If you read the comments, there appears to be a larger SIG working group discussion to evaluate how to properly handle Helm longer term, which I suspect this behavior will be addressed from that working group.

QuinnBast · 2022-04-05T14:54:50Z

@minnie-jeong-otsk I created a patch here and also published a release for this patch on my fork of kustomize (download the patched binary here). This patch works for us to pull oci:// registries. I will be submitting a MR to kustomize once my company allows us to sign the CLA.

Note that this release is NOT officially supported by kustomize.

natasha41575 · 2022-04-08T17:01:58Z

The Helm plugin builtin into kustomize is being migrated to a third party KRM extension, in which we will support OCI [issue]. I will try to prioritize the migration and make some progress for OCI support in Q2 given that there are a large number of requests for this.

/triage accepted

k8s-triage-robot · 2022-07-07T17:20:48Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

QuinnBast · 2022-07-13T15:05:54Z

/remove-lifecycle stale

QuinnBast · 2022-08-25T19:26:30Z

Based off of this comment, it appears that the helm chart inflation tool is supporting bug fixes and adding fields that are normally supported through helm template. That would make this issue and #4335 valid and would not be waiting for the new KRM functionality. Is there any priority on these two issues which already have a fix?

Edit: Never-mind... I didn't read the "we will not support OCI or private registries"... That is sad. I would consider being incompatible with Helm 3.8.0 features (which has been released for almost a year now) a bug...

elamcheliyana · 2022-09-16T13:03:13Z

I have tried the issue is still present. Does any one when they would consider to fix this problem or it going to be never?

igstbagusdharmaputra · 2022-10-18T03:35:59Z

will this problem be solved soon?

LelvProgrammer · 2022-11-07T09:43:42Z

Bump

85matthew · 2022-11-14T17:54:30Z

Any reason we can't reconsider one of the other pull requests that address this as a short term solution while the long term vision is still being planned?

https://github.com/kubernetes-sigs/kustomize/pull/4614/files

sthomson-wyn · 2022-12-23T14:58:59Z

There are PRs to fix this problem in kustomize. KRM is touted as the eventual solution, but has not had a commit in over 9 months...

antonydevanchi · 2023-01-02T11:22:23Z

@mikebz i'll dare you free blowjob if you close this shit, dude >_<

kosta · 2023-01-11T11:12:17Z

While it is stated under Long term support that

We will not add support for:

OCI registries

I would kindly ask you to reconsider this opinion, given that the fix actually is just +7-2 LOC and the krm function does not support this yet, either and is not production-ready it seems.

antonydevanchi · 2023-01-11T17:43:02Z

@kosta nevertheless it uses widely by Yandex.Cloud: site:cloud.yandex.ru "oci:" for example.

mikebz · 2023-01-11T21:10:52Z

I think there is a concern about backwards compatibility here, but the backwards compatibility issue was a side effect of the previous implementation and not an advertised feature. My vote is to move forward with the proposed PR.

andrzejgorski · 2023-02-07T19:56:18Z

As another ugly workaround, I found the solution:
Running kustomize with --helm-command ~/tmp/sanitized_helm.sh

where ~/tmp/sanitized_helm.sh is:

#!/bin/bash
if [[ $@ = pull* ]]; then
    # If the command is `helm pull (..)` skips --repo flag and chartName
    # from command line args to make helm pull run

    # For explanation:
    # https://github.com/kubernetes-sigs/kustomize/issues/4381
    arr=(${@//--repo/});  # Skipping --repo
    args="${arr[@]:0:5} ${arr[@]:6}";  # Skipping chartName
else
    args="$@"
fi
cmd="/usr/bin/helm --registry-config ~/.config/helm/registry/config.json $args"
echo "Running '$cmd' " >> /tmp/helm-debug
eval $cmd

SachithKasthuriarachchi · 2023-04-17T11:41:56Z

any update on this issue? this will be a great feature for kustomize and it will expand kustomize's adaptations

diegoluisi · 2023-06-21T18:38:57Z

Up!

QuinnBast · 2023-06-26T21:41:20Z

No, it's not in yet.

This MR is not against the actual kustomize repository; it will only get in after #5167 is approved by @natasha41575 and a release is created that contains the MR

annasong20 · 2023-09-10T07:25:12Z

@natasha41575 I'd like to check my understanding of this issue.

Initially, we planned to support this patch in the KRM Helm function, which would've had increased feature-freedom because of its independence from Kustomize. When progress on KRM functions slowed, we limited HelmChartInflationGenerator changes to lower maintenance costs.

Just to confirm, we are still open to this patch on HelmChartInflationGenerator due to its popularity?

annasong20 · 2023-09-10T07:32:20Z

@natasha41575 @QuinnBast or anyone else on this thread: Stemming from my lack of understanding of Helm, is Helm's inability to pull charts with the --repo flag a bug on their end? By adding the changes in #5167,

are we patching a Helm bug or
is the --repo behavior intentional and the kustomize: Add support for OCI based helm repos #5167 changes help Kustomize HelmChartInflationGenerator reflect new Helm behavior in relation to OCI

@natasha41575 If this is a Helm bug, I assume we're patching it in Kustomize instead of Helm as a short-term workaround for users?

natasha41575 · 2023-09-15T21:59:26Z

@annasong20 discussed offline - we will add support in kustomize for oci registries by having the different commands running that helm provides for oci-based charts.

pratik705 · 2024-06-10T14:03:07Z

is this fixed? I still cant use OCI registry with kustomize

mrclrchtr · 2024-06-10T17:33:32Z

Yes it is working. I use it like this::

helmCharts:
  - name: spegel
    repo: oci://ghcr.io/spegel-org/helm-charts
    releaseName: spegel
    namespace: spegel
    version: v0.0.23
    valuesFile: values.yaml

A few days ago I thought it wasn't working either, until I found out that I had entered the wrong repo URL 🤦 When you do that, you get an auth error that sends you on the wrong track.

QuinnBast added the kind/bug Categorizes issue or PR as related to a bug. label Jan 11, 2022

k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jan 11, 2022

QuinnBast changed the title ~~helmCharts repo field doesn't work with OCI protocol~~ helmCharts repo field doesn't work with OCI registry Jan 11, 2022

natasha41575 added the area/helm label Jan 11, 2022

QuinnBast mentioned this issue Feb 11, 2022

Error when fetching helm chart with 'kustomize build --enable-helm' #4451

Closed

QuinnBast mentioned this issue Mar 22, 2022

Support helm authentication for helm chart repositories. #4335

Closed

natasha41575 mentioned this issue Apr 8, 2022

OCI support for the render-helm-chart function kubernetes-sigs/krm-functions-registry#11

Closed

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 8, 2022

mikebz mentioned this issue May 2, 2022

OCI support in Helm builtin #4614

Closed

3 tasks

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 7, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 13, 2022

PG2000 mentioned this issue Jul 18, 2022

Support helm charts instead of kustomize as part of bootstrap command argoproj-labs/argocd-autopilot#38

Open

jkroepke mentioned this issue May 10, 2023

kustomize: Add support for OCI based helm repos #5167

Merged

cjyar mentioned this issue Sep 13, 2023

Add support for kustomize deployment bank-vaults/vault-operator#188

Open

k8s-ci-robot closed this as completed in #5167 Oct 17, 2023

JanGrosse mentioned this issue Nov 20, 2023

Not able to render kustomization file using helmCharts with oci:// repository url. akuity/kargo-render#222

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`helmCharts` repo field doesn't work with OCI registry #4381

`helmCharts` repo field doesn't work with OCI registry #4381

QuinnBast commented Jan 11, 2022 •

edited

Loading

KlavsKlavsen commented Jan 27, 2022 •

edited

Loading

QuinnBast commented Feb 9, 2022

QuinnBast commented Mar 22, 2022 •

edited

Loading

tis-rpage commented Mar 28, 2022

minnie-jeong-otsk commented Apr 5, 2022

tis-rpage commented Apr 5, 2022

QuinnBast commented Apr 5, 2022 •

edited

Loading

natasha41575 commented Apr 8, 2022 •

edited

Loading

k8s-triage-robot commented Jul 7, 2022

QuinnBast commented Jul 13, 2022

QuinnBast commented Aug 25, 2022 •

edited

Loading

elamcheliyana commented Sep 16, 2022

igstbagusdharmaputra commented Oct 18, 2022

LelvProgrammer commented Nov 7, 2022

85matthew commented Nov 14, 2022

sthomson-wyn commented Dec 23, 2022

antonydevanchi commented Jan 2, 2023 •

edited

Loading

kosta commented Jan 11, 2023 •

edited

Loading

antonydevanchi commented Jan 11, 2023

mikebz commented Jan 11, 2023

andrzejgorski commented Feb 7, 2023 •

edited

Loading

SachithKasthuriarachchi commented Apr 17, 2023

diegoluisi commented Jun 21, 2023

QuinnBast commented Jun 26, 2023

annasong20 commented Sep 10, 2023 •

edited

Loading

annasong20 commented Sep 10, 2023

natasha41575 commented Sep 15, 2023

pratik705 commented Jun 10, 2024

mrclrchtr commented Jun 10, 2024

helmCharts repo field doesn't work with OCI registry #4381

helmCharts repo field doesn't work with OCI registry #4381

Comments

QuinnBast commented Jan 11, 2022 • edited Loading

KlavsKlavsen commented Jan 27, 2022 • edited Loading

QuinnBast commented Feb 9, 2022

QuinnBast commented Mar 22, 2022 • edited Loading

tis-rpage commented Mar 28, 2022

minnie-jeong-otsk commented Apr 5, 2022

tis-rpage commented Apr 5, 2022

QuinnBast commented Apr 5, 2022 • edited Loading

natasha41575 commented Apr 8, 2022 • edited Loading

k8s-triage-robot commented Jul 7, 2022

QuinnBast commented Jul 13, 2022

QuinnBast commented Aug 25, 2022 • edited Loading

elamcheliyana commented Sep 16, 2022

igstbagusdharmaputra commented Oct 18, 2022

LelvProgrammer commented Nov 7, 2022

85matthew commented Nov 14, 2022

sthomson-wyn commented Dec 23, 2022

antonydevanchi commented Jan 2, 2023 • edited Loading

kosta commented Jan 11, 2023 • edited Loading

antonydevanchi commented Jan 11, 2023

mikebz commented Jan 11, 2023

andrzejgorski commented Feb 7, 2023 • edited Loading

SachithKasthuriarachchi commented Apr 17, 2023

diegoluisi commented Jun 21, 2023

QuinnBast commented Jun 26, 2023

annasong20 commented Sep 10, 2023 • edited Loading

annasong20 commented Sep 10, 2023

natasha41575 commented Sep 15, 2023

pratik705 commented Jun 10, 2024

mrclrchtr commented Jun 10, 2024

`helmCharts` repo field doesn't work with OCI registry #4381

`helmCharts` repo field doesn't work with OCI registry #4381

QuinnBast commented Jan 11, 2022 •

edited

Loading

KlavsKlavsen commented Jan 27, 2022 •

edited

Loading

QuinnBast commented Mar 22, 2022 •

edited

Loading

QuinnBast commented Apr 5, 2022 •

edited

Loading

natasha41575 commented Apr 8, 2022 •

edited

Loading

QuinnBast commented Aug 25, 2022 •

edited

Loading

antonydevanchi commented Jan 2, 2023 •

edited

Loading

kosta commented Jan 11, 2023 •

edited

Loading

andrzejgorski commented Feb 7, 2023 •

edited

Loading

annasong20 commented Sep 10, 2023 •

edited

Loading