seeking architectural advice building: ws client / dll implementation / threads

[davehorton]

I am looking for a bit of advice on how to best use lws in an application that is characterized by the following requirements:

1. The application acts as a ws client only.
2. The client is has both high bandwidth and low-latency (audio and video being streamed) requirements
3. My code - which is responsible for creating and managing the websocket connections -- must be in a dll that is loaded into a 3rd party application. I am not the author of the 3rd party application, but I do have access to and could patch the code if need be. The 3rd party application provides a standard api for loading user-provided dlls such as the ones I will create.
4. I will have probably 5-8 separate dlls that will be loaded and operating simultaneously in this manner, each making ws connections to a server and sending data.
5. The application will be running on a multi-core server and I want to make use of threads if possible to ensure that I dont saturate one core sending data while leaving others starved.

Now, I have already created an initial working implementation, so my question is really whether I can tune this to improve performance, and whether anything I have done is potentially "out of bounds" or dangerous, so let me explain the current implementation below.

In the current implementation, in my dll code when the dll loads I have an initialization function that creates a new lws thread. I have a single context and the lws thread simply loops:

  context = lws_create_context(&info);
  if (!context) {
    return false;
  }

  int n;
  do {
    n = lws_service(context, 0);
  } while (n >= 0 && !stopFlag);

Therefore, when I load, say, 5 dlls they all create a context and an lws thread and run indepdendently as above. And generally everything seems to work ok. However, I do notice issues with lws logging and I suppose it is because each dll when loaded is calling lws_set_log_level() so that is getting called multiple times, each with a different logging function and potentially log levels. So that is one issue.

A second issue is that I do encounter the issue where the lws thread can saturate one cpu core and leave others idle, so I would like to understand the best approach for spreading the work that lws is doing across multiple cores. Could I, for instance, simply create n contexts and n lws_service threads in my code and then when creating a new websocket connection just call lws_cancel_service on a randomly-selected context to spread the workload? Is that reasonable, or does that lead to problems ? (I did try that before I think I did experience some crashes).

To summarize some of the questions I would love advice on:

1. Given the requirements, can I keep all my lws code in the dlls as I have done, or am I better off patching the main application to host some or all of the lws api calls?
2. Given I will have multiple dlls creating ws client connections and being loaded and unloaded dynamically, is there anything I should keep in mind that could cause clashes or crashes I should avoid?
3. Given the requirements, what would be the best general approach to scale multiple lws threads across n cpus?

Thanks in advance for any help, and thanks for all the great work on this awesome project.