Support message sessions in Azure Service Bus Scaler

[worldspawn]

Use-Case

I have a large workload that is bound to a single session id. As a session can only be processed from a single listener (because it locks the session) scaling according to the number of messages in the queue is not... useful.

I've just added 620 messages to a queue all with the same session id and keda has scaled up the service to whopping 16 instances. But unfortunately 15 instances are doing nothing and 1 is working hard.

I'm not sure if you can interrogate the number of active sessions but if you can then...

Specification

• Allow to specify the desired number of desired active sessions and scale based on that instead of the number of messages in a queue.

[tomkerkhove]

That's a good scenario to cover. Are you using a specific scaler or more in general?

With Azure Service Bus you'd use sessions indeed, I'm just wondering how KEDA could know when to scale and when not?

[worldspawn]

I'm using the azure-servicebus trigger currently. I can't see any appropriate method on the REST api that would do this but "Service Bus Explorer" has the ability to produce a list of sessions for a queue (somehow).

[worldspawn]

Seems to use this method: https://docs.microsoft.com/en-us/dotnet/api/microsoft.servicebus.messaging.subscriptionclient.getmessagesessions?view=azure-dotnet#Microsoft_ServiceBus_Messaging_SubscriptionClient_GetMessageSessions

[tomkerkhove]

Reasonable ask. So if there would be a useSessions: true opt-in which would scale based on metrics and sessions that would help you?

[worldspawn]

Yeah that would be great. In this particular case I wouldn't be interested in scaling based on the number of messages at all, only the number of sessions. Because of how the .net client manages concurrency I cannot imagine ever needing to scale based on both queue length and session count at the same time. I think its as simple as - if its a session enabled queue i want to scale based on sessions, otherwise queue count.

[tomkerkhove]

Makes sense, thanks for bringing this up!

[davidallyoung]

Hi everyone, just to chime in, we also have a similar use case for sessions. Some context on how we use them:

We utilize the sessions to break up workloads per tenant and help facilitate "fast lanes". It's set up in such a way that some tenants could have more sessions than others, etc. If KEDA could scale based on sessions, it would definitely simplify and make more reliable our scaling strategy.

[MCGPPeters]

Somewhat connected to this issue.... When downscaling, are receiver instances that are locked taken into account? E.g. are these instances kept alive until the session has completed and the lock has been removed?

[tomkerkhove]

No because KEDA is not aware of what instances are doing. It just tells Kubernetes to scale from x to y, and the scheduler will kill the required amount of pods.

[MCGPPeters]

That would be really nice to have, since long running sessions could theoretically span days / months ... Any way you could work around this?

[tomkerkhove]

The only way to achieve this is to change the application that's running so that when a shutdown is initiated it gracefully abandons the session for somebody else to continue later on.

[MCGPPeters]

Thanks. That is the strategy I am following atm indeed. Was hoping I could remove some code :)

[tomkerkhove]

WDYT @ahmelsayed?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[purijatin]

We have a similar use case. It would be great to have a working solution on this.

[tomkerkhove]

Are you willing to contribute this?

[raorugan]

Sure @tom ***@***.***>, will look into this and get back on the decision. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Tom Kerkhove ***@***.***> Sent: Friday, February 24, 2023 12:09:26 PM To: kedacore/keda ***@***.***> Cc: Ramya Oruganti ***@***.***>; Assign ***@***.***> Subject: Re: [kedacore/keda] Support message sessions in Azure Service Bus Scaler (#1479) Are you willing to contribute this? — Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkedacore%2Fkeda%2Fissues%2F1479%23issuecomment-1442873136&data=05%7C01%7CRamya.Oruganti%40microsoft.com%7C4c239ed1bf6a4bdc7e5108db1631dfd4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128175729225150%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JPri%2Fpg8epsHhMokwFyR5J3X4DcM7OhkhQYJtw5jshE%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAK4F7NYJFRK44V55MONBWVDWZBJR5ANCNFSM4VW2R2JA&data=05%7C01%7CRamya.Oruganti%40microsoft.com%7C4c239ed1bf6a4bdc7e5108db1631dfd4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638128175729225150%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BG0Dmig03NmyoPUv%2FzezTM%2FMT2Ktvqgp7Px160xamfs%3D&reserved=0>. You are receiving this because you were assigned.Message ID: ***@***.***>

[tomkerkhove]

Oh no need to @raorugan

[Krishnadas-KP]

@tomkerkhove Will this be implemented soon ?

[tomkerkhove]

It's looking for a contributor so if you are interested, let me know.

[b-dines]

We are also interested in this capability. I am not sure we have the skills/bandwidth to contribute unfortunately.

[christopherlombardiconfluence]

We also have an interest in this capability.

[miqm]

We're also interested in this feature.

[miqm]

@tomkerkhove Is this feature even possible to implement currently? I think that the Service Bus SDK for go doesn't provide option to get count of active sessions. There's option in C# and Java SDK (GetMessageSessions on SB client) but I don't see this in GO.

[tomkerkhove]

We have not looked in to it yet given nobody has proposed to implement it yet. If this is a known limitation, then it would be nice if you could file an issue with them!

[petemessina]

@tomkerkhove I can take a look at the change and see if I can move it forward

[aionic]

@petemessina / @tomkerkhove - I would be interested in helping as well

[tomkerkhove]

That would be wonderful! @petemessina / @aionic who wants to pick this up?

[petemessina]

I can pick it up

[sujesharukil]

Any movement on this feature?

[Mortana89]

Following here, we use sessions to guarantee FIFO so queue based scaling as is is indeed not useful, would love to have this!

[miqm]

@tomkerkhove - I found a ticket from last year where .NET SDK team discussed on Service Bus sessions. It seems that this feature needs to have not only SDK for GO support but also the Service Bus API: Azure/azure-sdk-for-net#20569 (comment)

For scaling scenarios this feature on service bus API is kind of crucial. Do you think you could influence Service Bus team to bring this on their roadmap and provide such data?

[miqm]

Hello,

For those who are still waiting for the native support and want to try something different - I have started an open-source project with external scaler that leverages AMQP command to get count of sessions: https://github.com/miqm/session-scaler. Feel free to try it out. I will gradually try to improve it i.e. add workload identity support. All feedback (even the negative, as long as is constructive ) is appreciated!