MDAEngine threading mechanism

[jacopoabramo]

As mentioned #395, I had another question: is there a specific thread infrastructure in the MDAEngine? Is everything single-threaded? Could one spawn multiple threads of execution? If so, how?

[tlambert03]

as far as the existing MDAEngine that drives the CMMCore object, threading issues work like this:

• MDARunner itself is single threaded and fully synchronous (*see detail about callback handlers below)
• pymmcore_pus.mda.MDAEngine is also single threaded and fully synchronous (on the python side)
• on the C++ side, CMMCore absolutely uses threads and coordinates the activity of multiple devices, and so calls made by MDAEngine into the C++ code may indeed result in threaded behavior. In my opinion, this is actually a big bonus of using CMMCore. It's all implemented at a pretty low/performant level, so many of the synchronous calls we make from the python/pymmcore side of things will result in concurrent device behavior on the C++ side (though that too can be modified)
• most of the time, we invoke MDARunner.run() in a separate thread, so the MDA acquisition tends to take place in its own thread, and that's mostly so that we can use the main thread for GUI stuff. CMMCorePlus.run_mda is a convenience method that calls MDARunner.run() in a new thread

• in the MDARunner inside of pymmcore_plus, you may notice a mda_listeners_connected method, That is a thread that exists just to receive any frameReady signals that are emitted by the Runner. So basically, the runner executes an event, emits the payload as a signal, and some other thread somewhere is processing the data as it comes off, for example to write to disk, coordinate smart feedback, etc... That's sort of a side detail, but just wanted to list it since we're discussing threads.

[tlambert03]

Could one spawn multiple threads of execution? If so, how?

I'm not entirely sure what you're picturing here, but I think it's safe to say "no". Can you clarify what you're picturing?

[jacopoabramo]

Something like having two concurrent MDA sequences running from the same engine, e.g. (or i.e.? i never know the difference): imagining that in an ideal world there's an extra argument called thread passable to core.mda.run, and retaking your two lines of code from the previous discussion:

seq1 = MyFancySequence() # yeah this should be actually an useq-schema object but bear with me
seq2 = MyFancySequence(args = different) # again, bear with me

core.mda.set_engine(custom_engine)
core.mda.run(seq1, thread = thread1)
core.mda.run(seq2, thread = thread2)

But from your original answer I'm assuming that this does not happen. Also as it stands I'm guessing it would be difficult to run two sequences which may involve the same device

[tlambert03]

from your original answer I'm assuming that this does not happen. Also as it stands I'm guessing it would be difficult to run two sequences which may involve the same device

exactly. sharing device control to execute two independent sequences across two different threads is not something we support... it does seem incredibly complex once we bring actual hardware into the picture, and to be honest, on first glance looks like a YAGNI feature to me. In any pattern where you are controlling literal singletons (hardware devices) then you need some sort of master scheduler, and the MDARunner kind of is that master scheduler.

but if you've seen such a thing actually done in the wild I'd love to know! (for example, concurrently executing two different multi-dimensional acquisitions using the same microscope driven by two different clients)

[tlambert03]

The queue pattern described here might be relevant here, if you're thinking of some sort of concurrent multi-client single server setup (where there might be multiple remote clients controlling a single microscope).

from queue import Queue
from pymmcore_plus import CMMCorePlus
from useq import MDAEvent

core = CMMCorePlus()
core.loadSystemConfiguration()

q = Queue()                    # create the queue
STOP = object()                # any object can serve as the sentinel
q_iterator = iter(q.get, STOP) # create the queue-backed iterable

# start the acquisition in a separate thread
core.run_mda(q_iterator)

# now we can put events into the queue
# according to whatever logic we want:
q.put(MDAEvent(index={'t': 0}, exposure=20))
q.put(MDAEvent(index={'t': 1}, exposure=40))

# ... and eventually stop the acquisition
q.put(STOP)

in that case, there is still just a single MDARunner that dispatches MDAEvents to an Engine. And that runner could absolutely receive commands (in the form of MDAEvents) from multiple different clients concurrently, but I don't think you'd want to start runner.run twice

[jacopoabramo]

but if you've seen such a thing actually done in the wild I'd love to know!

Well I'm pondering on a solution for having a better control of one of our setups (an OSPIM) to add an extra channel, but it's a convoluted problem on it's own. Fut for now this is all thinking work that I'm doing in order to better know the capabilities of the MDAEngine since I remember stumbling upon it but never gave too much tought to it back in the day.

[tlambert03]

pondering on a solution for having a better control of one of our setups (an OSPIM) to add an extra channel, but it's a convoluted problem on it's own.

always interested to hear the actual problem or use case if you care to share