Autoscaling not working with Keda

[nidiculageorge]

Report

The autoscaling is not working as expected after the upgrade to version 2.16.1

Expected Behavior

The autoscaling should work

Actual Behavior

Autoscaling not working

Steps to Reproduce the Problem

kedaoperatorerror08012024.txt

2.When we send message from service bus the scaling should happen but its not happening

Logs from KEDA operator

2025-01-08T16:22:17Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "pdfgenerator-autoscaler", "error": "subscription r112atstriggermessageprocessing doesn't exist in topic r112atstriggermessageprocessing"}
2025-01-08T16:22:17Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "orchestratorservice-autoscaler", "error": "subscription orchestratorallevents doesn't exist in topic allevents"}
2025-01-08T16:22:17Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "pilmrservice-autoscaler", "error": "Get "": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:17Z ERROR scale_handler error getting scale decision {"scaledObject.Namespace": "", "scaledObject.Name": "pilmrservice-autoscaler", "scaler": "azureServiceBusScaler", "error": "Get "": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:17Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "largepdfgenerator-autoscaler", "error": "Get "": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:17Z ERROR scale_handler error getting scale decision {"scaledObject.Namespace": "a3services-uat-icanr111", "scaledObject.Name": "largepdfgenerator-autoscaler", "scaler": "azureServiceBusScaler", "error": "Get "https://.servicebus url/?api-version=2021-05": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "pilmrservice-autoscaler", "error": "Get "https://.servicebus url/necltr111createpilmr/Subscriptions/necltr111createpilmr?api-version=2021-05": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR scale_handler error getting scale decision {"scaledObject.Namespace": "", "scaledObject.Name": "pilmrservice-autoscaler", "scaler": "azureServiceBusScaler", "error": "Get "https://.servicebus url/necltr111createpilmr/Subscriptions/necltr111createpilmr?api-version=2021-05": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "a3services-uat-caqa1r111", "name": "messagequeueservice-autoscaler", "error": "Get "https://.servicebus url/?api-version=2021-05": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "largepdfgenerator-autoscaler", "error": "subscription r112largeatstriggermessageprocessing doesn't exist in topic r112largeatstriggermessageprocessing"}
2025-01-08T16:22:18Z ERROR scale_handler error getting scale decision {"scaledObject.Namespace": "", "scaledObject.Name": "largepdfgenerator-autoscaler", "scaler": "azureServiceBusScaler", "error": "subscription r112largeatstriggermessageprocessing doesn't exist in topic r112largeatstriggermessageprocessing"}
2025-01-08T16:22:18Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "pdfgenerator-autoscaler", "error": "Get "https://sb?api-version=2021-05\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR scale_handler error getting scale decision {"scaledObject.Namespace": "", "scaledObject.Name": "pdfgenerator-autoscaler", "scaler": "azureServiceBusScaler", "error": "Get "https://sb?api-version=2021-05\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "a3services-dev-secr111", "name": "documentconverterservice-autoscaler", "error": "Get "https://sb-": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR scale_handler error getting scale decision {"scaledObject.Namespace": "", "scaledObject.Name": "documentconverterservice-autoscaler", "scaler": "azureServiceBusScaler", "error": "Get "https://sb": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}

==================

KEDA Version

2.16.1

Kubernetes Version

1.30

Platform

Microsoft Azure

Scaler Details

Azure Service bUS

Anything else?

Can see a error like this also

Can see the following in Keda Scalerfailed event

context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[nidiculageorge]

Keda Metrics Logs as below

kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
meta.helm.sh/release-name: keda
meta.helm.sh/release-namespace: keda
creationTimestamp: "2021-09-23T04:46:43Z"
labels:
app.kubernetes.io/component: operator
app.kubernetes.io/instance: keda
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
app.kubernetes.io/part-of: keda-operator
app.kubernetes.io/version: 2.16.1
helm.sh/chart: keda-2.16.1
name: v1beta1.external.metrics.k8s.io
resourceVersion: "4674562812"
uid: b7a324d0-77b8-4c7d-bdec-b79d3e560db0
spec:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFoTVJBd0RnWURWUVFLRXdkTFJVUkIKVDFK1lEVlFRREV3UkxSVVJCTUI0WERUSXpNRGd6TURFeE1qa3lNMW9YRFRNek1EZ3lOekV5TWpreQpNMW93SVRFUU1BNEdBMVVFQ2hNSFMwVkVRVTlTUnpFTk1Bc0dBMVVFQXhNRVMwVkVRVRRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHR5Sjl1SU5SSlhHRnA0MEFjeGZnemRLSlVyMTFrQjlFTG8KZjBLNUpwMWhLRFk2RXdhcDBJMUpxOEN2MWdPZWN3UWXc3doaThSeHEydDFrajNLaDFpVUFYYmFTcAoxNTEydVlhdWIycG5JN2N0YmV1em9lRFc4YUJ4NGZlVEYzL1M5aFFBZjBvYWFFRk5UUzUvajl1ZUlYMGVSWHBICkJrZlNFanNPS3c5ZXlaDUyNHZqcmptaXVOS0hOSHZGZk5BaU9zYzBjRFRvUEovc0RqOVp0emZzNTIKSVYyWi9TUWliOEQ0RWhKcERIZmo1aC9qaDBsM2RCWTFFSnpSNVBvUnRXNTF4NzZHRk5uMHFYMUtpOGE4zFTcEJPZTlzZEhRV0poK3Z3Ui9WS3ovSVZjeXEyUUp6U2dQS3FmVFcwMTRjbGNCQzhDQXdFQUFhTlRNRkV3CkRnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZZSME9CQllFRkpoNFpmNDMKYk5GTzhJbEhabGFKbzZjaHVXamtNQThHQTFVZEVRUUlNQWFDQkV0RlJFRXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQURUQ3hORTNKQTcyWSt2Ujk3MEUQlN1dVZqeFdvTC85dVZseXRRNHlMUXBlOVBJc2VlQ1lGClVWMURsNEJYOTlqV1Z1WEg3aXF0QVlpQjR4OWYzVjliZURpOFFib0hqaElIVUNxd3gzbTZLWllhcUhuQ1UyaTEKZFZzS0lc05lRnRaMVhMcXZ5aHdTdDVHeFRFenhKWkY3THBlTmFRTHV1ZXl4UXpSbUZrd1JCYjgrVQpCMSthcVVFK3NJMnpLTkxlM3NNNWlmTy8xTFZTQzBlc3lsckIwMDBwSUlRd1paRy8yaVcym9WUmtDCjFERG96YzQwSmg1THdpL2NaaHpDcTRHRXZMZnhzblZzV3E3OS9TbWEwMWpIaHpqQlpkOGJ4RE1lZERxZVNOU2cKZVdsOEJ4cnIvc3RPcGRUVTJiREIxZ2hYS1ZzPQotLS0tLJUSUZJQ0FURS0tLS0tCg==
group: external.metrics.k8s.io
groupPriorityMinimum: 100
service:
name: keda-operator-metrics-apiserver
namespace: keda
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:

• lastTransitionTime: "2025-01-09T08:14:51Z"
  message: all checks passed
  reason: Passed
  status: "True"
  type: Available

[nidiculageorge]

@JorTurFer thanks for the triage.As the ticket is being long time ,is there any updates that you can provide on this

[JorTurFer]

Could you share your ScaledObject (and your trigger authentication)? just remove the sensitive data please
From logs I see a lot of 401 errors

[nidiculageorge]

@JorTurFer

2025-01-08T16:22:18Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "a3services-uat-caqa1r111", "name": "messagequeueservice-autoscaler", "error": "Get "https://.servicebus url/?api-version=2021-05": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2025-01-08T16:22:18Z ERROR azure_servicebus_scaler error getting service bus entity length {"type": "ScaledObject", "namespace": "", "name": "largepdfgenerator-autoscaler", "error": "subscription r112largeatstriggermessageprocessing doesn't exist in topic r112largeatstriggermessageprocessing"}

Scaled object name : messagequeueservice-autoscaler

Trigger : azure-servicebus

[zroubalik]

This is something you should check:

subscription r112largeatstriggermessageprocessing doesn't exist in topic r112largeatstriggermessageprocessing"

[nidiculageorge]

@zroubalik does these errors cause failures in keda scaling ...we are fixing these issues after pulling the keda operator logs .

But these timeout issues are due to the above subscription missing