Ideas for selecting one remote connection in a list of many

[minsis]

Basically looking for ideas on how to create some kind of round robin type selection when connecting to remote servers.

In my example we have two remote servers that can do the exact same thing so it would be nice to be able to determine which one is available and maybe even which one might have more load than others. Would be similar to a load balancer just not as complex as one.

[nzlosh]

Is there a StackStorm context or are you asking about load balancing in general?

[minsis]

Looking for ideas on how to implement (mimic) load balancing on the StackStorm side not adding servers to an actual load balancer. I think its a kind of interesting idea to try on StackStorm somehow.

We have a commercial software (similar to st2) that lets you create a "pool" of hosts and then instead of executing on a specific host you execute the action on the pool. Then the software determines which node is best suited based on some simple rules. Basically round robin, ordered list, or least load.

What makes it easy for the commercial software is that each node is running an agent that reports back the info so there's no need for making a connection to the server first like st2 would have to. However, in our environment we run Zabbix which has agents so it might be easy to maybe execute Zabbix agent commands to get similar info.

The only idea I had was to create an action that would take in a list of hosts and a balancing type.

Something like:

1. Read list
2. Check if each node is even up
3. See how many actions are currently being executed on the node (not sure how to do this)
   • Depending on the rule set being used (e.g. in order) go to 4 or 6
4. Maybe gather some basic stats on the node (current load, free mem, etc)
5. Use step 3 and 4 to determine best node
6. Return the name of the node

Then this action could sit in front of other actions in a workflow.

[nzlosh]

If I understand you correctly, you're not talking about distributing workflow execution across a St2 cluster but to have workflows pick hosts to use for core.remote or similar actions.

To emulate the notion of a pool with resource consumption in mind, I'd use a service discover system to identify hosts that would be part of the pool. I'd use a graphing system to collect the resource usage of each node in the pool. I'd query these two sources of truth and filter the data with YAQL (or jinja if you prefer) to choose the least loaded host.

I recommend depending on external systems because they're built for the specific task in mind and will avoid adding a whole lot of complexity to St2 workflows.

[minsis]

How would you have StackStorm look internally to know how many current connections it has to a specific node?

[nzlosh]

You wouldn't, the solution I presented is purely workflow based, they don't have access to St2 internal state.

[minsis]

Well there is access to the datastore, maybe the action can do something with writing the number of processes. I guess in this case you would need two actions one at the start and one at end of the workflow in order to mark that. Seems a bit messy.

I've created a custom Zabbix check that will check all the concurrent threads for the stackstorm user.

[minsis]

In the end we decided to put the nodes behind a LB and let it figure out which node to use. Then we implemented polices on on the pack to limit the number of executions as we had over 50 rules all needing to queue at the same time.

Still, it would be neat to see some sort of pack that could handle this from StackStorm. I started working on one, but I dont have time to deal with experiential things right now.