vulnerabilityreports in few namespaces are not getting created #2202

santoshkarp3 · 2024-07-29T13:30:32Z

What steps did you take and what happened:

vulnerabilityreports in few namespaces are not getting created

What did you expect to happen:
vulnerabilityreports in few namespaces are not getting created getting folloing error in opretor logs -

{"level":"error","ts":"2024-07-29T13:20:35Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: repository name error (registry-rat.prd.harbor.vodafone.com/vf-gks-trivy-artifacts/aquasecurity/trivy-checks:0:0): could not parse reference: registry-rat.prd.harbor.vodafone.com/vf-gks-trivy-artifacts/aquasecurity/trivy-checks:0:0","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*NodeReconciler).SetupWithManager.(*NodeReconciler).reconcileNodes.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/node.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"} {"level":"error","ts":"2024-07-29T13:20:35Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"158.179.249.157"},"namespace":"","name":"158.179.249.157","reconcileID":"69fe2006-6174-4368-8186-89468256d251","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"} {"level":"error","ts":"2024-07-29T13:21:23Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: repository name error (registry-rat.prd.harbor.vodafone.com/vf-gks-trivy-artifacts/aquasecurity/trivy-checks:0:0): could not parse reference: registry-rat.prd.harbor.vodafone.com/vf-gks-trivy-artifacts/aquasecurity/trivy-checks:0:0","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*NodeReconciler).SetupWithManager.(*NodeReconciler).reconcileNodes.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/node.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"} {"level":"error","ts":"2024-07-29T13:21:23Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"158.179.249.157"},"namespace":"","name":"158.179.249.157","reconcileID":"8f3930fc-fb29-4d55-a579-9470fe374987","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

Environment:

Trivy-Operator version (use trivy-operator version): 0.21.1
Kubernetes version (use kubectl version): 1.29.21 ( OKE )
OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Oracle Linux Server 7.9

The text was updated successfully, but these errors were encountered:

santoshkarp3 · 2024-07-29T13:36:19Z

I have found bug - #2101

so I tried to use latest help chart "0.24.1" but still getting this errors

santoshkarp3 · 2024-07-29T13:37:11Z

@chen-keinan Could you please help me . Thanks In advance
some more logs "
{"level":"error","ts":"2024-07-29T07:17:33Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-77d5b4fd79","container":"trivy-operator","status.reason":"Error","status.message":"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"} {"level":"error","ts":"2024-07-29T10:22:01Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-77d5b4fd79","container":"trivy-operator","status.reason":"Error","status.message":"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"} {"level":"error","ts":"2024-07-29T11:22:40Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-6596cf9c85","container":"nvidia-gpu-device-plugin","status.reason":"OOMKilled","status.message":"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"} {"level":"error","ts":"2024-07-29T13:31:43Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-76854fd8f6","container":"grafana-sc-dashboard","status.reason":"Error","status.message":"","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

Hacks4Snacks · 2024-07-29T14:12:24Z

@santoshkarp3

In the second set of logs, there are multiple generic errors stemming from scan job containers, but there is one with an out of memory error. If you are using the default resource limits for the scan job containers, I recommend trying to increase resource allocation and reviewing. Separately, it seems there is some potential for error logging improvement.

santoshkarp3 · 2024-07-29T14:13:39Z

captured error form one of scan-vulnerabilityreport-59467869d4-bzp8rpod

santoshkarp3 · 2024-07-29T14:15:56Z

@Hacks4Snacks
Hello , I have increased 250/1000M now . Pls ignore OOO error .
Pls help me to fix the other like stacktrace . that cloud be one reason not get vulnerabilityreport from some namespaces

Hacks4Snacks · 2024-07-29T14:16:25Z

captured error form one of scan-vulnerabilityreport-59467869d4-bzp8rpod

Ah, this looks related to: #2191

santoshkarp3 · 2024-07-30T09:38:13Z

Hello @Hacks4Snacks @chen-keinan
pls help me on this issue .
Thanks in advance

santoshkarp3 · 2024-07-30T13:01:53Z

reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-55bb576f45","container":"grafana-sc-datasources","status.reason":"Error","status.message":"","stacktrace
Anyone faced this issue ?

And infraassessmentreports.and clusterinfraassessmentreports , clusterconfigauditreports. and not getting generated

santoshkarp3 · 2024-08-28T12:25:23Z

error 👍{ "level": "error", "ts": "2024-08-28T12:18:41Z", "logger": "reconciler.scan job", "msg": "Scan job container", "job": "trivy-system/scan-vulnerabilityreport-7b5d749d7d", "container": "archival-container", "status.reason": "Error", "status.message": "2024-08-28T12:18:35Z\tFATAL\tFatal error\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:aa035a908267cc209122798c35be4ae479508bf1e5e6bf3720a1c90d7d96246f): walk error: failed to process the file: failed to analyze file: failed to analyze usr/java/jdk-17.0.9/jmods/java.base.jmod: unable to open usr/java/jdk-17.0.9/jmods/java.base.jmod: failed to open: unable to read the file: unexpected EOF\n", "stacktrace": "github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).completedContainers\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:353\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).SetupWithManager.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:80\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222" }

github-actions · 2024-10-28T00:27:51Z

This issue is stale because it has been labeled with inactivity.

dyadik9616 · 2024-11-27T16:18:51Z

@afdesk getting this error

controllerKind: "Node"
error: "creating job: no compliance commands found"
level: "error"
msg: "Reconciler error"
name: "gke-*"
reconcileID: "b5e9d1c1-da10-4766-b7fd-82bac06db451"
stacktrace: "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"

any suggestion how to fix it?

afdesk · 2024-11-28T14:33:47Z

@afdesk getting this error

controllerKind: "Node"
error: "creating job: no compliance commands found"
level: "error"
msg: "Reconciler error"
name: "gke-*"
reconcileID: "b5e9d1c1-da10-4766-b7fd-82bac06db451"
stacktrace: "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"

any suggestion how to fix it?

what is the version of trivy-operator?
could you try with the latest - v0.23.0?

mdnfiras · 2025-01-13T00:18:50Z

@afdesk getting this error

controllerKind: "Node"
error: "creating job: no compliance commands found"
level: "error"
msg: "Reconciler error"
name: "gke-*"
reconcileID: "b5e9d1c1-da10-4766-b7fd-82bac06db451"
stacktrace: "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"

any suggestion how to fix it?

also getting the same error, and running version 0.23.0 on gke

piotr-muzyka · 2025-01-14T09:12:41Z

+1 I am also running v0.23.0 on GKE cluster and encountering the same no compliance commands found error.

navpreet-securitas · 2025-01-14T09:54:29Z

same error happens when running v0.23.0 on AKS cluster. Here are logs with DEBUG enabled

DEBUG   node-reconciler Getting node from cache {"node": {"name":"aks-gcpanp01-03662548-vmss00000f"}}
2025-01-14T09:42:17Z    DEBUG   node-reconciler Checking whether cluster Infra assessments report exists        {"node": {"name":"aks-gcpanp01-03662548-vmss00000f"}}
2025-01-14T09:42:17Z    DEBUG   node-reconciler Checking whether Node info collector job have been scheduled    {"node": {"name":"aks-gcpanp01-03662548-vmss00000f"}}
2025-01-14T09:42:17Z    DEBUG   node-reconciler Checking node collector jobs limit      {"node": {"name":"aks-gcpanp01-03662548-vmss00000f"}, "count": 0, "limit": 10}
2025-01-14T09:42:18Z    DEBUG   node-reconciler Scheduling Node collector job   {"node": {"name":"aks-gcpanp01-03662548-vmss00000f"}}
2025-01-14T09:42:18Z    ERROR   Reconciler error        {"controller": "node", "controllerGroup": "", "controllerKind": "Node", "Node": {"name":"aks-gcpanp01-03662548-vmss00000f"}, "na
mespace": "", "name": "aks-gcpanp01-03662548-vmss00000f", "reconcileID": "bb36e284-6d68-4517-b2aa-7b1a378aec88", "error": "creating job: no compliance commands found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224

Hence nodeCollector pods / job doesn't get created(though vulnerability scan jobs get schedules and results are there). Here are helm values related to nodeCollector:

   nodeCollector:
      useNodeSelector: false
      tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
       - key: app
          operator: Equal
          value: gcp
          effect: NoSchedule

afdesk · 2025-01-14T09:58:36Z

@piotr-muzyka @navpreet-securitas thanks a lot for the reports!
I'm checking it

ChanduReddy123 · 2025-01-15T12:08:59Z

I was able to fix this issue by excluding the nodes

nodeCollector:
      useNodeSelector: false
      excludeNodes: kubernetes.io/arch=amd64

now I am no longer getting the error no compliance commands found

in my case I am only looking for image scans and not the nodescan so I am ok with the global node exclusion

afdesk · 2025-01-30T09:33:32Z

Hi guys! sorry for long delay with response.

Finally, I managed to reproduce a similar issue on my local kind cluster with four nodes.

logs

{"level":"error","ts":"2025-01-30T06:24:21Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: Get \"https://mirror.gcr.io/v2/aquasec/trivy-checks/blobs/sha256:d42567de6666044bbf31833e2948c24262d857d9fdf40f695ac2267192f3dba3\": dial tcp: lookup mirror.gcr.io on 10.96.0.10:53: server misbehaving","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*NodeReconciler).SetupWithManager.(*NodeReconciler).reconcileNodes.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/node.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:21Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"cve-kind-control-plane"},"namespace":"","name":"cve-kind-control-plane","reconcileID":"48e5a606-cf8a-4fa1-b4e3-4a56adf86575","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:34Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: Get \"https://mirror.gcr.io/v2/aquasec/trivy-checks/blobs/sha256:d42567de6666044bbf31833e2948c24262d857d9fdf40f695ac2267192f3dba3\": dial tcp: lookup mirror.gcr.io on 10.96.0.10:53: server misbehaving","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*NodeReconciler).SetupWithManager.(*NodeReconciler).reconcileNodes.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/node.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:34Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"cve-kind-control-plane"},"namespace":"","name":"cve-kind-control-plane","reconcileID":"9cd3b0d9-7507-4f66-a6f1-3352397a1441","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:45Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: Get \"https://mirror.gcr.io/v2/aquasec/trivy-checks/blobs/sha256:d42567de6666044bbf31833e2948c24262d857d9fdf40f695ac2267192f3dba3\": dial tcp 142.251.1.82:443: connect: connection refused","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*NodeReconciler).SetupWithManager.(*NodeReconciler).reconcileNodes.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/node.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:45Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"cve-kind-control-plane"},"namespace":"","name":"cve-kind-control-plane","reconcileID":"f1feb618-63ff-4eca-81e5-bf367d95133b","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:45Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: Get \"https://mirror.gcr.io/v2/aquasec/trivy-checks/blobs/sha256:d42567de6666044bbf31833e2948c24262d857d9fdf40f695ac2267192f3dba3\": dial tcp 142.251.1.82:443: connect: connection refused","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*NodeReconciler).SetupWithManager.(*NodeReconciler).reconcileNodes.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/node.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:45Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"cve-kind-control-plane"},"namespace":"","name":"cve-kind-control-plane","reconcileID":"e1442c75-775b-45e6-8742-160854009b0c","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:45Z","logger":"policyLoader.Get misconfig bundle policies","msg":"failed to load policies","error":"failed to download policies: failed to download built-in policies: download error: oci download error: failed to fetch the layer: Get \"https://mirror.gcr.io/v2/aquasec/trivy-checks/blobs/sha256:d42567de6666044bbf31833e2948c24262d857d9fdf40f695ac2267192f3dba3\": dial tcp 142.251.1.82:443: connect: connection refused","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/policy.(*policyLoader).GetPoliciesAndBundlePath\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/policy/loader.go:63\ngithub.com/aquasecurity/trivy-operator/pkg/configauditreport/controller.(*NodeReconciler).SetupWithManager.(*NodeReconciler).reconcileNodes.func5\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/configauditreport/controller/node.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.TypedFunc[...].Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-01-30T06:24:45Z","msg":"Reconciler error","controller":"node","controllerGroup":"","controllerKind":"Node","Node":{"name":"cve-kind-control-plane"},"namespace":"","name":"cve-kind-control-plane","reconcileID":"99dde0ec-3113-430e-9c4c-469ebf754fc1","error":"creating job: no compliance commands found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224"}

After reviewing the logs, it appears that this message contains two independent issues.

The first issue is related to the error from the first message:
"error":"failed to download policies: failed to download built-in policies: download error: repository name error (registry-rat.prd.harbor.vodafone.com/vf-gks-trivy-artifacts/aquasecurity/trivy-checks:0:0): could not parse reference: registry-rat.prd.harbor.vodafone.com/vf-gks-trivy-artifacts/aquasecurity/trivy-checks:0:0"

In this case, it is recommended to first verify that the trivy-checks database in Harbor matches the current versions of trivy-operator and trivy.
Moreover, the next Trivy release will include a mirroring option, which may help in managing policy downloads more efficiently.

The second issue is related to the error: "creating job: no compliance commands found".
This is most likely caused by the trivy-operator lacking permissions to run the node-collector job on control-plane nodes.
As a solution, as mentioned in the comment above, you can either set the excludeNodes value or use tolerations to allow node-collector to run only on the appropriate nodes.

in any case, I don't think there are bugs here now, so I'd close this issue.

Please feel free to create a new issue if any errors will appear! thanks for the reports!

santoshkarp3 added the kind/bug Categorizes issue or PR as related to a bug. label Jul 29, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed. label Oct 28, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 17, 2024

afdesk removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed. label Nov 18, 2024

afdesk reopened this Nov 18, 2024

simar7 assigned afdesk Dec 3, 2024

simar7 added this to the v0.24.0 milestone Dec 4, 2024

afdesk closed this as completed Jan 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vulnerabilityreports in few namespaces are not getting created #2202

vulnerabilityreports in few namespaces are not getting created #2202

santoshkarp3 commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024 •

edited

Loading

Hacks4Snacks commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024 •

edited

Loading

Hacks4Snacks commented Jul 29, 2024

santoshkarp3 commented Jul 30, 2024

santoshkarp3 commented Jul 30, 2024 •

edited

Loading

santoshkarp3 commented Aug 28, 2024

github-actions bot commented Oct 28, 2024

dyadik9616 commented Nov 27, 2024 •

edited

Loading

afdesk commented Nov 28, 2024

mdnfiras commented Jan 13, 2025

piotr-muzyka commented Jan 14, 2025 •

edited

Loading

navpreet-securitas commented Jan 14, 2025 •

edited

Loading

afdesk commented Jan 14, 2025

ChanduReddy123 commented Jan 15, 2025

afdesk commented Jan 30, 2025

vulnerabilityreports in few namespaces are not getting created #2202

vulnerabilityreports in few namespaces are not getting created #2202

Comments

santoshkarp3 commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024 • edited Loading

Hacks4Snacks commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024

santoshkarp3 commented Jul 29, 2024 • edited Loading

Hacks4Snacks commented Jul 29, 2024

santoshkarp3 commented Jul 30, 2024

santoshkarp3 commented Jul 30, 2024 • edited Loading

santoshkarp3 commented Aug 28, 2024

github-actions bot commented Oct 28, 2024

dyadik9616 commented Nov 27, 2024 • edited Loading

afdesk commented Nov 28, 2024

mdnfiras commented Jan 13, 2025

piotr-muzyka commented Jan 14, 2025 • edited Loading

navpreet-securitas commented Jan 14, 2025 • edited Loading

afdesk commented Jan 14, 2025

ChanduReddy123 commented Jan 15, 2025

afdesk commented Jan 30, 2025

santoshkarp3 commented Jul 29, 2024 •

edited

Loading

santoshkarp3 commented Jul 29, 2024 •

edited

Loading

santoshkarp3 commented Jul 30, 2024 •

edited

Loading

dyadik9616 commented Nov 27, 2024 •

edited

Loading

piotr-muzyka commented Jan 14, 2025 •

edited

Loading

navpreet-securitas commented Jan 14, 2025 •

edited

Loading