Application fails to synchronize - error: plugin sidecar failed. error generating manifests in cmp

[jgagnon44]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

We have a Kubernetes cluster where we use ArgoCD to manage and synchronize updates. We recently upgraded ArgoCD to version v2.5.15 and also did the work to switch from using plugins to sidecars. Everything appears to be in good condition, with one exception. An application fails to synchronize with the message:

rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c "helm dependency build\n"` failed exit status 1: Error: no cached repository for helm-manager-ad0ea17f2a357d9a8e390ad4d1a7baf91c56ff6891dd88e9dd273134a29256c1 found. (try 'helm repo update'): open /home/argocd/.cache/helm/repository/helm-manager-ad0ea17f2a357d9a8e390ad4d1a7baf91c56ff6891dd88e9dd273134a29256c1-index.yaml: no such file or directory

The application is deployed via a Helm chart. The chart does not have any external dependencies.

To Reproduce

Not sure what to tell you here. As mentioned above, it is an internal application that is failing to synchronize with ArgoCD. This is happening within our company internal network.

Expected behavior

For the application to be successfully synchronized when updates are committed to our GitLab repository being monitored by ArgoCD.

Screenshots

Version

Paste the output from `argocd version` here.

argocd: v2.5.15+64a7bb8
  BuildDate: 2023-03-16T22:34:19Z
  GitCommit: 64a7bb8f02238777ec45752d7655dba505eed889
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64

Logs

Paste any relevant application logs here.

I do not know if it would be helpful, but I am including what I think are relevant files.

The main Chart.yaml:

apiVersion: v2
type: application
name: prometheus-metrics-ref
description: Provides an HTML page listing all Prometheus metrics and rules.
version: 1.0.1

and the associated values.yaml:

application:
  image:
    repository: harbor.hulk.beast-code.com/phactory-images/prometheus-metrics-and-rules
    tag: latest
  env:
    port: 8080
    prometheusApiUrl: ""
  ingress:
    domain: hulk.beast-code.com
    subdomain: ""
  resources:
    requests:
      cpu: 125m
      memory: 512Mi
    limits:
      cpu: 1000m
      memory: 2Gi

A K8s cluster-specific Chart.yaml to deploy the app to the cluster:

apiVersion: v2
name: prometheus-metrics-ref-argocd
description: Prometheus metrics and rules reference app.
type: application
version: 1.0.0

dependencies:
  - name: prometheus-metrics-ref
    version: 1.0.1
    repository: https://harbor.hulk.beast-code.com/chartrepo/library

and its values.yaml:

prometheus-metrics-ref:
  nameOverride: arc-prometheus-metrics-ref
  application:
    env:
      prometheusApiUrl: https://prometheus.nnpi.beast-code.com/api/v1
    ingress:
      domain: nnpi.beast-code.com
      subdomain: ""

Our cmp-plugin:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  avp-kustomize.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-kustomize
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - find
            - "."
            - -name
            - kustomization.yaml
      generate:
        command:
          - sh
          - "-c"
          - "kustomize build . | argocd-vault-plugin generate -"
      lockRepo: false
  avp-helm.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      init:
        command:
          - sh
          - "-c"
          - |
            helm dependency build
      generate:
        command:
          - sh
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME --include-crds -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_helm_args} . |
            argocd-vault-plugin generate -
      lockRepo: false
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "."
      lockRepo: false

[lukaszkubisiak]

same here

[jgagnon44]

There may be an underlying issue. An unrelated application deployed to a separate K8s cluster (managed with ArgoCD) seems to be suffering a similar problem. The error seen in ArgoCD for this app is slightly different, but the nested error seems to be the same:

I have found existing tickets that seem to be very relevant. helm/helm#11369 and helm/helm#11372.

[DesmondH0]

Hi guys, I got a similar issue but I am in argocd 2.7.1 and the helm repository in the Chart.yaml is private which required username and password to login.

I did set the repository correctly in ArgoCD level [ref]
But seems the plugin sidecar from repoServer do not pick that up and throwing the same error as OP.
If I go into the plugin sidecar and add the helm repo manually, it would work and generated the app manifest as expected.

Wondering if this behaviour is expected, at this moment I am thinking to embed the helm repo in a customise plugin image as a workaround.

[alexvaque]

same here

EDITED, In my case it is caused by the argo-vault-plugin so once I uninstalled the argo-vault-plugin the problem has gone

[crenshaw-dev]

Wondering if this behaviour is expected

It's expected. We can't safely send all repo credentials to the sidecar, because it's likely more credentials than 1) the app needs and 2) the app's project has access to. We need to design a way, ideally, for the sidecar to request exactly what credentials it needs and then for Argo CD to supply those if and only if the project has access to them. Someone in Slack a few days ago mentioned they'd write a proposal.

Related: #8820

[wouter2397]

Any updates on this issue?
We are currently also using argocd-vault-plugin with a helm dependency to a private repo.
Currently we are unable to pull those charts from our registry server due to the fact that credentials are not shared to the sidecar containers.

[thesuperzapper]

You can get helm dependency update (or helm dependency build) to succeed by simply adding the repos from the Chart.yaml to the repo cache using helm repo add.

This can be automated by parsing the result of helm dependency list:

# add the helm repos for the chart dependencies
helm dependency list --max-col-width 10000 "./path/to/chart" | awk 'NR>1 {print $1,$3}' | while read -r name url; do
    if [[ -n "$name" && -n "$url" ]]; then
        helm repo add "$name" "$url" --insecure-skip-tls-verify
    fi
done

# update the helm dependencies
helm dependency update "./path/to/chart"

Note, I added --insecure-skip-tls-verify because it could not verify valid HTTPS/TLS certs for some reason.

[wouter2397]

@thesuperzapper Thanks for commenting on this issue. I used the same workaround to add the required Helm repositories during the init phase.

I would like to see here a definitive solution to propagate the credentials from the Argo CD container into the CMP containers.

[thesuperzapper]

Hi everyone, I have found a few issues with my solution from #13539 (comment):

1. The need for --insecure-skip-tls-verify was because TLS ca-certificates is not included in busybox, so I now use docker.io/buildpack-deps:bookworm-curl for my plugin sidecar.
2. The helm dependency list command can return URLs like file:// and oci:// so we need to filter to http/https.
3. If the URL of a repo changes, helm repo add fails to update it unless you set --force-update

Here is my new solution:

# add the helm repos for the chart dependencies
helm dependency list --max-col-width 10000 "./path/to/chart" | awk 'NR>1 {print $1,$3}' | while read -r name url; do
    if [[ -n "$name" && -n "$url" && "$url" =~ "^https?://" ]]; then
        helm repo add "$name" "$url" --force-update
    fi
done

# update the helm dependencies
helm dependency update "./path/to/chart"

[msobkowiak-olx]

Bumped into the same issue when migrated from 2.7.3 to 2.8.x

What seems to work for us is to remove requirements.yaml and move any dependencies to Chart.yaml...

[albertomfb]

Hi there,

Maybe I'm missing something, but I can't understand the workaround mentioned of using helm repo add
(it still needs the --username --password parameters to work with secured helm repositories...)

The conclusion at the moment is that you cannot use the avp plugin with helm repos with credentials ?
(despite registering them in argo, no credentials are provided to sidecar as mentioned before in this ticket)
Any other ideas?

Thanks in advance.