question regarding "on-the-fly" processing

[tlambert03]

Originally posted by @raulbola in #293 (comment)

I have a question regarding "on-the-fly" processing. We are developing a very fast OS-SIM microscope and I have implemented the image reconstruction algorithms such that the inference time is so fast that allows me to have a real time prview of the sample. Up to now I was using my own Labview/Matlab GUIs and control programs with Live Preview modes that continously acquire the set of N raw images, combine them using OS-SIM algoritm and display the result in the Live window (such as the Apotome mode in Zeiss SIM).

May I ask if there is a way to do so with napari micromanger?Something like the on-the-fly processor of Micromanager, but changing the Frame Averaging or Maximum projection by our OS-SIM algorithm?

[tlambert03]

Hi @raulbola, (opening a new topic for visibility, since this will likely be a common question)

There's not formal framework in place at the moment, but the basis is there and is a very active area of development at the moment. Here's a high level overview. Note that napari-micromanager itself is a rather thin wrapper at the moment for pymmcore-widgets and pymmcore-plus, so most of this answer will point to those libraries

• in napari-micromanager, when you run an experiment in the MDA tab, you are constructing a useq.MDASequence object (using a pymmcore-widgets.MDAWidget)
• that object is passed to the pymmcore_plus.mda.MDARunner.run() method, which is responsible for iterating over the individual events in the sequence and passing each one to the pymmcore_plus.mda.MDAEngine which sets up and executes the events. (documentation overview here)
• After each frame, the runner emits a frameReady event with the new image, the useq.MDAEvent that triggered it, and additional metadata. (more about events here)

... this is where it gets interesting/flexible. Anyone can register callbacks with the runner.

• napari-micromanager itself is simply a callback that registers a callback for those events and shows them in the napari viewer (by adding each frame to the zarr array underlying the napari image layer).
• you could also register file-saving callbacks like ImageSequenceWriter, or the OMETiffWriter and OMEZarrWriter (in development).
• Those writers use the general pattern of mda_listeners_connected:

  class MyHandler:
      def frameReady(self, ary, event, meta):
          ...
  
  seq = useq.MDASequence(...)
  with mda_listeners_connected(SlowHandler()):
      core.mda.run(seq)

So, the real crux of the "on-the-fly" question here is the ordering/relationship of those callbacks. Does that make sense?

The first question you need to ask is whether you need to modify the acquisition parameters themself? to change how the future images are acquired?

• If so, you'll (currently) need to use a CustomAcquisition Engine, or one of the strategies presented in the Event Driven Acquisition docs. For napari-micromanager, those will still require some manual coding (minimally, you'll need to call core.mda.set_engine(YourEngine())
• if not, then the question is whether you want the processing to simply change how the image is presented in the napari viewer or whether you want the processing to happen before saving to disk. And then inject that processing into the appropriate frameReady callback chain accordingly. (There's no way to do that easily at the moment, but that's where this is all headed).

I'll also link this image.sc post which is a related question.

hope that helps give a lay-of-the land. Comments/strategies/proposals definitely welcome

[raulbola]

Hi @tlambert03 that makes total sense. Thank you for the detailed explanation, now I have a better understanding of the napari-micromanager structure.

At this moment I do not need to change the acquisition parameters, so I think the best solution now is to just process the acquired stack of images just before being displayed in napari or before saving to disk.

I don't know the specific order of events for the Micromanager On-The-Fly processing engine, so that why I was asking.

Actually this question came up to me when reading the pycromanager documentation, their acquisition_superclass.py already supports this option. You can specify the image_process_fn.

[tlambert03]

You can specify the image_process_fn.

Yep, there is not yet an equivalent on the pymmcore-plus side, largely because saving itself is also less well defined. This is all part of the saving PRs mentioned above, and still a work in progress