ksops + argocd + helm - plugin doesn't seem to be active/run

[conor-nsurely]

Hi all,

I have installed Argocd using helm, with the helm values file used to configure the init container which installs ksops to Argocd .

I am using the helm values from here

And all the values seem to be present in the configmap / deployments in the cluster.

I am using Argocd v2.9 and viaductoss/ksops:v4.3.1

In the argo-repo-server logs I can see

│ install-ksops Installing KSOPS...                                                                                                                                                                                                              │
│ install-ksops Done.

And if I start a shell on the argo-repo-server, "kustomize" and "ksops" are present on the path and seem to be installed correctly.

When I add an encrypted secret to Git, Argo fails to deploy saying
one or more objects failed to apply, reason: Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 3 (retried 5 times).

It seems like the plugin is not running and Argocd is just trying to deploy the encrypted secret directly.

Where am I going wrong?

( I have tried $XDG_CONFIG_HOME = /.config also to no avail)

[dosubot]

🤖

Hello @conor-nsurely! 👋

I'm Dosu, a friendly bot 🤖 here to assist while we wait for a human maintainer. I can help you solve bugs, answer questions, and guide you to become a contributor. Let's crack this nut together, shall we? 🌰

Based on the information you've provided and similar issues I found in the KSOPS repository, it seems like the issue might be related to the ArgoCD configuration not being set up correctly to use the KSOPS plugin.

Here are a few things you can check:

1. Ensure that the KSOPS plugin is correctly installed in the ArgoCD repo server. You can do this by using an init container to install the KSOPS plugin and Kustomize into an emptyDir volume, and then mounting this volume into the ArgoCD repo server container. This is shown in the "KSOPS Repo Sever Patch" section of the README.

2. Make sure that the ArgoCD server is configured to enable Kustomize alpha plugins and exec functionality. This can be done by setting the kustomize.buildOptions field in the ArgoCD ConfigMap to "--enable-alpha-plugins --enable-exec". This is shown in the "Argo CD Helm Chart with Custom Tooling" section of the README.

3. If you're using AWS or GCP KMS for encryption, ensure that the necessary credentials to decrypt the secrets are included in the ArgoCD repo server container's environment variables. This is mentioned in the "KSOPS Repo Sever Patch" section of the README.

4. If you're still having issues, you can try building a custom ArgoCD Docker image that includes the KSOPS plugin and Kustomize, as shown in the "Custom Argo CD w/ KSOPS Dockerfile" section of the README.

In addition to these, I found two similar issues in the KSOPS repository that might be helpful:

1. KSOPS not working with ArgoCD - no error which was last updated on January 04, 2024. The suggested solution is to ensure the -enable-exec flag is present in kustomize.buildOptions.

2. KSOPS Config With Argo Values.yaml - Unable to find plugin root which was last updated on October 24, 2023. The suggested solution is to set the XDG_CONFIG_HOME environment variable correctly and ensure the volume mounts are configured properly.

I hope this helps! If you're still having issues, could you provide more details about your setup? Specifically, it would be helpful to see the exact configuration you're using for the ArgoCD repo server and the KSOPS plugin.

Sources

• KSOPS not working with ArgoCD - no error
• KSOPS Config With Argo Values.yaml - Unable to find plugin root
• README.md

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[devstein]

Hey @conor-nsurely thanks for making a detailed issue. At least we know that KSOPS is installed correctly.

1. Can you verify that kustomize.buildOptions: "--enable-alpha-plugins --enable-exec" is set on the ArgoCD config map?

2. If it is, are you able to share more information about your K8s manifests and directory structure?

[conor-nsurely]

Hey @devstein

1. It is enabled but I did notice there's also a flag disabling exec. Not sure where that has come from, maybe a default in Helm

data:
  admin.enabled: "true"
  application.instanceLabelKey: argocd.argoproj.io/instance
  exec.enabled: "false"
  kustomize.buildOptions: --enable-alpha-plugins --enable-exec
  server.rbac.log.enforce.enable: "false"

I've chaged exec.enabled to true, but there doesn't seem to be a difference.

---

2. Here's the secret that Argo is failing to apply

apiVersion: v1
data:
    cache_secret: ENC[AES256_GCM,data:d0Yg4P60Vi5AcwxlSJ/oPw==,iv:VwcckTKniflsOFzBLCzDDgbJB8W63MAknlrx52afaX8=,tag:oF3Xoj5aVFdR7D74UJO9yQ==,type:str]
kind: Secret
metadata:
    name: 123-secret
    namespace: default
type: Opaque
sops:
    kms: []
    gcp_kms:
        - resource_id: projects/dev/locations/global/keyRings/ksops-test/cryptoKeys/gke
          created_at: "2024-01-15T16:07:55Z"
          enc: CiQAR1OX8+1vLMju1VB94xq2gJkVki4eI6847X9NEXv3cHPMGRUSSQAFusxYIaFO2vNbWnaXrIx+u1xaH9LfQfxYqcwXtzrrU1h+rliLX6MWJuWfClj3g04JaA8iElW3EGy6Us5w/WvfRDfZfjEGHB0=
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2024-01-15T16:07:55Z"
    mac: ENC[AES256_GCM,data:Zr+bXPfbkq+lQAr9y9Ybue6Qqf+sNPE05hpnGtzF0tdUlnAWOVMFzCGzQZoZnqQlUxO/SQiPR1ncJxWy8ab4LDY5k8IoI+HygDK6dC1igXB+dJawTGGfRKhCtM5tOOlxIYG6rYRg43SlNKzNE7C6f4m90qW5xsDT+NSefG9jUn4=,iv:atOxnkFhkRU/8uGpTNi2YdcX/N/M7oynpTR0WogH7MA=,tag:ARnoyZVsXyVVncVSCqYswQ==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.8.1

I have the secret in a a top level directory by itself. I have put other yaml files in there as a sanity check and they all deploy just fine.

---

3. I assume I'm generating the secret incorrectly.

Do I need to apply anything else to kubernetes?`Like the KSOPS kustomize Generator from step 5 of the general instructions.

kind: ksops
metadata:
  name: example-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops

---

4. Is it possible to use Workflow Identity federation to retrieve the GCP key or does the key have to be passed via an environment variable? I assume there would be another error though if failing to decrypt was the sole issue.

Really appreciate the help!

[conor-nsurely]

@devstein Is there anymore information I can provide to make the issue clearer?

[hugopicq]

Hi ! I am running into the same problem here, any news?

[valkiriaaquatica]

Same here

[7434-atul]

HI conor-nsurely i am also facing the same issue, i have deployed the argocd using instal.yaml, created the init containers and followed all the steps for the repo-server-deployment and then tried with docker image as well but still no awail, the error message which you are getting i am getting the same error while using helm.

[devstein]

Hi @7434-atul can you share your kustomization.yaml? It's possible that the upstream helm chart changed and the instructions are stale

[devstein]

and can you check the value of the argo-cm ConfigMap?

[7434-atul]

hi @devstein , i have added following detail in the cm as well
kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
I cannot see any error as well when the application is deployed , it is deploying the other object but not secrets.yaml which is present in the root directory

[7434-atul]

@devstein i have used following application to deploy the application,
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: test
namespace: argocd
spec:
project: test
source:
repoURL: *******************
path: test/app
targetRevision: test
helm:
valueFiles:
- secrets://secrets.yaml
# - secrets+age-import:///helm-secrets-private-keys/key.txt?templates/secrets.yaml
- values-test.yaml
# - secrets+age-import:///helm-secrets-private-keys/key.txt?secrets.yaml
destination:
server: https://kubernetes.default.svc
namespace: test
ignoreDifferences:

• group: "apps"
  kind: "Deployment"
  jsonPointers:
  • /spec/replicas
    syncPolicy:
    automated:
    prune: true
    selfHeal: true
    syncOptions:
    • CreateNamespace=true
    • ApplyOutOfSyncOnly=true
    • RespectIgnoreDifferences=true