feat: add MDAEvent.slm_image

[tlambert03]

closes #202

• MDAEvent.slm_image is a new pydantic object SLMImage with two attributes (at the moment), data and device
• data is np.typing.ArrayLike (anything that can be case to a numpy array). But for simplicity with pydantic, we type define it as Any.
• if device is None, it's up to the backend to pick the "current device"
• SLMImage is itself castable to a numpy array via np.asarray(event.slm_image)
• For now, I'm leaving all of the additional details that @hinderling mentioned in Add SLM mask to MDAEvent #202 (comment) (such as casting to different data types and dealing with device-specific on values) to backends like pymmcore[-plus].

data = [[0, 0], [1, 1]]

# directly passing data
event = MDAEvent(slm_image=data)
assert event.slm_image is not None

# we can cast SLMIamge to a numpy array
assert isinstance(np.asarray(event.slm_image), np.ndarray)
np.testing.assert_array_equal(event.slm_image, np.array(data))

# variant that also specifies device label
event2 = MDAEvent(slm_image={"data": data, "device": "SLM"})
assert event2.slm_image is not None
np.testing.assert_array_equal(event2.slm_image, np.array(data))
assert event2.slm_image.device == "SLM"

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.87%. Comparing base (d5f0525) to head (1fb7434).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #204      +/-   ##
==========================================
+ Coverage   93.80%   93.87%   +0.06%     
==========================================
  Files          18       18              
  Lines        1243     1257      +14     
==========================================
+ Hits         1166     1180      +14     
  Misses         77       77              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[hinderling]

Wow that was quick 😍🚀 Started working on it, but I'll abandon those efforts then haha. Happy to test though, we're setting up a new microscope with pymmcore, so the timing is perfect!

Some questions I had:

• How to deal with the SLM exposure time? We use the light source as shutter, but maybe there are people that would want to copy the camera exposure time to the SLM (or set it independently).
• If slm_device != none AND slm_image==none, it would be nice to default to all pixels ON, so the DMD is used as a normal light source. But maybe this is also specific to our setup, and would lead to difficulties when it's unclear what value an ON pixel requires.

[tlambert03]

Started working on it, but I'll abandon those efforts then haha.

oh i'm sorry! i didn't realize ... I should have let you do it.

How to deal with the SLM exposure time? We use the light source as shutter, but maybe there are people that would want to copy the camera exposure time to the SLM (or set it independently).

good question. Since we have an independent SLMImage object here, we can certainly add a new exposure: float | None = None attribute, where None means "just use the MDAEvent.exposure". Sound good?

If slm_device != none AND slm_image==none, it would be nice to default to all pixels ON, so the DMD is used as a normal light source. But maybe this is also specific to our setup, and would lead to difficulties when it's unclear what value an ON pixel requires.

yeah, that is probably for pymmcore-plus to worry about. But here in useq-schema, we can certainly add documentation that specifies what a backend SHOULD do when they encounter a null slm_image.data?

[hinderling]

No what you have is much better of course haha so I'm super happy!

we can certainly add a new exposure: float | None = None attribute, where None means "just use the MDAEvent.exposure"

If None, I would expect the DMD to just switch on the pixels on indefinitely, until SetSLMImage and DisplaySLMImage are called again? Otherwise if both are e.g. 20ms and are not synchronized by TTL, there could be a significant offset.
But now I'm unsure how it's actually handled. For sure I had some issues with one DMD model where I had to set the SLMExposureTime to the max time of 2 minutes for liveview (then turn it on again after two minutes).

that is probably for pymmcore-plus to worry about. But here in useq-schema, we can certainly add documentation that specifies what a backend SHOULD do

perfect!

[tlambert03]

If None, I would expect the DMD to just switch on the pixels on indefinitely, until SetSLMImage and DisplaySLMImage are called again?

here too I'm a little concerned about overloading semantics (on the schema side) too much. Perhaps if the MDAEvent wants to indicate "turn on all pixels", there should be an event with MDAEvent(slm_image=True), where a single boolean item (as opposed to a 2D array) implies "upload all on/off". But, I think we could actually play with that a bit on the pymmcore-plus side before coming back here to update the schema "formally". sound good?

[tlambert03]

If None, I would expect the DMD to just switch on the pixels on indefinitely,

oh sorry, you were talking about exposure right? I might have missed that. In any case ... let's start here, try to implement it on the pymmcore-plus side (would love your help there!) and then circle back if we find we need additional fields/restrictions over here.

[hinderling]

Ah yes, on the exposure time! I imagine the DMD mirrors like a x/y stage, an event sets a new position and if there's no new command it remains in that position (if exposure is None).

slm_image = True for all on sounds perfect, but this again would require solving the issue with knowing the ON pixel value for all SLM devices, leaving this to the backend sounds great. Would be fantastic if pymmcore knew how to cast boolean arrays for all devices.

Great plan! Let me know how I can best help. Started adding code to the setup sequence, on our setup it would look like this:

if event.slm_image is not None:
            slm_dev = self._mmc.getSLMDevice()
            if slm_dev:
                try:
                    self._mmc.setSLMImage(slm_dev, event.slm_image)
                    self._mmc.displaySLMImage(slm_dev)
                except Exception as e:
                    logger.warning("Failed to set SLM image. %s", e)

This of course doesn't deal with the exposure times (and possibly sequenceable DMDs?)

[tlambert03]

ok, in we go!