ScaledObject Fails to resolve Auth params from cluster scoped ClusterTriggerAuth resource #6316

sreeboppana · 2024-11-06T23:40:59Z

Report

Keda version: keda:2.14.0
ghcr.io/kedacore/keda:2.14.0
kubenetes version: 1.29
Keda Trigger Type: Kafka

ScaledObject Fails [namespace scoped] to resolve Auth params from cluster scoped ClusterTriggerAuth resource

Expected Behavior

Keda ScaledObject ideally should have read the credentials from the ClusterTriggerAuth resource and authenticate with the KAfka API.

Actual Behavior

2024-11-06T16:49:43Z  ERROR scale_handler error resolving auth params {"type": "ScaledObject", "namespace": "my-app-namespace", "name": "my-app-microservice", "triggerIndex": 4, "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
  /workspace/pkg/scaling/scalers_builder.go:99
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
  /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
  /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
  /workspace/pkg/scaling/scale_handler.go:611
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:243
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

2024-11-06T16:49:43Z  ERROR scale_handler error getting state of scaledObject {"scaledObject.Namespace": "my-app-namespace", "scaledObject.Name": "my-app-microservice", "error": "error getting scalers cache error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:245
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

Steps to Reproduce the Problem

STEP-1 Ensured that k8s Secret - keda-kafka-secret created within the keda namespace.

% kubectl get secret keda-kafka-secret -oyaml -nkeda

apiVersion: v1
data:
  keda-kafka-api-key: KAFKA-API-KEY
  keda-kafka-api-secret: KAFKA-API-SECRET
kind: Secret
type: Opaque
metadata:
  creationTimestamp: "2024-10-31T08:06:59Z"
  labels:
    secrets-store.csi.k8s.io/managed: "true"
  managedFields:
  - apiVersion: v1
  name: keda-kafka-secret
  namespace: keda

STEP-2. Created a cluster scoped ClusterTriggerAuthentication resource & refer to the
secret from STEP-2 which is keda namespace scoped.

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-kafka-clustertriggerauth
spec:
  secretTargetRef:
  - parameter: keda-kafka-api-key
    name: keda-kafka-secret
    key: keda-kafka-api-key
  - parameter: keda-kafka-api-secret
    name: keda-kafka-secret
    key: keda-kafka-api-secret

STEP-3. Create a ScaledObject within my-app-namespace namespace referring to the above cluster scoped ClusterTriggerAuthentication to read the credentials & authenticate with Kafka (kafka trigger).

---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: my-app-namespace
spec:
  scaleTargetRef:
    name: my-demo-app-deployment
  pollingInterval: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafkaHost:9092
      consumerGroup: my-consumer-group
      topic: kafka-demo-topic
      lagThreshold: "30"
    authenticationRef:
      name: keda-kafka-clustertriggerauth
      kind: ClusterTriggerAuthentication

STEP-4 Just describe keda-operator pod (informational purpose).

% kubectl describe po keda-operator-5b48dfb687-b7xcx -n keda

Name:         keda-operator-5b48dfb687-b7xcx
Namespace:    keda
Priority:     0
Node:         ip-100-66-9-70.ec2.internal/100.66.9.70
Start Time:   Tue, 05 Nov 2024 07:40:10 -0500
Labels:       app=keda-operator
              app.kubernetes.io/component=operator
              app.kubernetes.io/instance=keda-operator
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keda-operator
              app.kubernetes.io/part-of=keda-operator
              app.kubernetes.io/version=2.14.0
              helm.sh/chart=keda-2.14.2
              name=keda-operator
              pod-template-hash=5b48dfb687
Annotations:  kubectl.kubernetes.io/restartedAt: 2024-10-31T09:06:59+01:00
Status:       Running
IP:           172.22.199.48
IPs:
  IP:           172.22.199.48
Controlled By:  ReplicaSet/keda-operator-5b48dfb687
Containers:
  keda-operator:
    Container ID:  containerd://94334ff68a5dab55f2c32a1f10a54230eaa2139ff2654fa20c858cf5b31c5dc8
    Image:         ghcr.io/kedacore/keda:2.14.0
    Image ID:      ghcr.io/kedacore/keda@sha256:c74847dd7c2a62d6a3ad0631208dab53264cec4eb0fc5ac3f41d4852ee363bf6
    Ports:         8080/TCP, 9666/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /keda
    Args:
      --leader-elect
      --disable-compression=true
      --zap-log-level=info
      --zap-encoder=console
      --zap-time-encoding=rfc3339
      --cert-dir=/certs
      --enable-cert-rotation=true
      --cert-secret-name=kedaorg-certs
      --operator-service-name=keda-operator
      --metrics-server-service-name=keda-operator-metrics-apiserver
      --webhooks-service-name=keda-admission-webhooks
      --k8s-cluster-name=kubernetes-default
      --k8s-cluster-domain=cluster.local
      --enable-prometheus-metrics=true
      --metrics-bind-address=:8080
    State:          Running
      Started:      Tue, 05 Nov 2024 07:40:13 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  8Gi
    Requests:
      cpu:      1
      memory:   8Gi
    Liveness:   http-get http://:8081/healthz delay=25s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=20s timeout=1s period=3s #success=1 #failure=3
    Environment:
      WATCH_NAMESPACE:              
      POD_NAME:                     keda-operator-5b48dfb687-b7xcx (v1:metadata.name)
      POD_NAMESPACE:                keda (v1:metadata.namespace)
      OPERATOR_NAME:                keda-operator
      KEDA_HTTP_DEFAULT_TIMEOUT:    3000
      KEDA_HTTP_MIN_TLS_VERSION:    TLS12
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::12345679:role/keda-operator-iam-role-us-east-1
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /certs from certificates (ro)
      /mnt/kafka-secret from kafka-secret (ro)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m9vcm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kedaorg-certs
    Optional:    true
  kafka-secret:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:            
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=keda-kafka-secretproviderclass
  kube-api-access-m9vcm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Logs from KEDA operator

2024-11-06T16:49:43Z  ERROR scale_handler error resolving auth params {"type": "ScaledObject", "namespace": "my-app-namespace", "name": "my-app-microservice", "triggerIndex": 4, "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
  /workspace/pkg/scaling/scalers_builder.go:99
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
  /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
  /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
  /workspace/pkg/scaling/scale_handler.go:611
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:243
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

2024-11-06T16:49:43Z  ERROR scale_handler error getting state of scaledObject {"scaledObject.Namespace": "my-app-namespace", "scaledObject.Name": "my-app-microservice", "error": "error getting scalers cache error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:245
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

KEDA Version

2.14.0

Kubernetes Version

1.29

Platform

Amazon Web Services

Scaler Details

Kafka

Anything else?

No response

The text was updated successfully, but these errors were encountered:

sreeboppana · 2024-11-11T18:25:29Z

@JorTurFer can you please advise on this please ?

apoole-q6cyber · 2024-11-20T17:35:11Z

This might be a question for stackoverflow

JorTurFer · 2024-11-23T22:27:49Z

Sorry for the delay.
Is your secret in KEDA's namespace? When you use a ClusterTriggerAuthentication, the secrets can be only in the namespace where KEDA is.

sreeboppana · 2024-12-02T16:38:09Z

@JorTurFer yes the secret is within the keda namespace please refer to STEP-1 above.

JorTurFer · 2024-12-15T16:59:09Z

Hello
Sorry for the slow response, my life is quite chaotic last months :(

The problem is with your ClusterTriggerAuthentication

spec:
  secretTargetRef:
  - parameter: keda-kafka-api-key
    name: keda-kafka-secret
    key: keda-kafka-api-key
  - parameter: keda-kafka-api-secret
    name: keda-kafka-secret
    key: keda-kafka-api-secret

the field 'parameter' is the parameter of the trigger, for example:

spec:
  secretTargetRef:
  - parameter: username
    name: keda-kafka-secret
    key: keda-kafka-api-key
  - parameter: password
    name: keda-kafka-secret
    key: keda-kafka-api-secret

(ofc, replace with the properties that you are using)

sreeboppana · 2025-01-03T17:38:34Z

thanks @JorTurFer yes it fixed it. Replacing the following clustertriggerauth resource worked for us while authenticating with confluent-kafka.

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-kafka-clustertriggerauth
spec:
  secretTargetRef:
  - parameter: username
    name: keda-kafka-secret
    key: username
  - parameter: password
    name: keda-kafka-secret
    key: password

I am good to close this issue.

sreeboppana added the bug Something isn't working label Nov 6, 2024

keda-automation added this to Roadmap - KEDA Core Nov 6, 2024

github-project-automation bot moved this to To Triage in Roadmap - KEDA Core Nov 6, 2024

sreeboppana mentioned this issue Jan 3, 2025

AWS SecretsManager TriggerAuth Unable to get secret #6249

Open

JorTurFer closed this as completed Jan 17, 2025

github-project-automation bot moved this from To Triage to Ready To Ship in Roadmap - KEDA Core Jan 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ScaledObject Fails to resolve Auth params from cluster scoped ClusterTriggerAuth resource #6316

ScaledObject Fails to resolve Auth params from cluster scoped ClusterTriggerAuth resource #6316

sreeboppana commented Nov 6, 2024

sreeboppana commented Nov 11, 2024

apoole-q6cyber commented Nov 20, 2024

JorTurFer commented Nov 23, 2024

sreeboppana commented Dec 2, 2024

JorTurFer commented Dec 15, 2024

sreeboppana commented Jan 3, 2025 •

edited

Loading

ScaledObject Fails to resolve Auth params from cluster scoped ClusterTriggerAuth resource #6316

ScaledObject Fails to resolve Auth params from cluster scoped ClusterTriggerAuth resource #6316

Comments

sreeboppana commented Nov 6, 2024

Report

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Logs from KEDA operator

KEDA Version

Kubernetes Version

Platform

Scaler Details

Anything else?

sreeboppana commented Nov 11, 2024

apoole-q6cyber commented Nov 20, 2024

JorTurFer commented Nov 23, 2024

sreeboppana commented Dec 2, 2024

JorTurFer commented Dec 15, 2024

sreeboppana commented Jan 3, 2025 • edited Loading

sreeboppana commented Jan 3, 2025 •

edited

Loading