DM-43340: Add exposure pairing for full array mode

[jmeyers314]

PR adds two new subtasks:
ExposurePairer : automatically pairs intra/extra exposures based on focusz and proximity in time/space/rotation.
TablePairer : pairs intra/extra exposures based on a pre-ingested table of exposure IDs.

Also included are scripts to ingest above table, and modifications to CutoutDonutsScienceSensorTask to use the pairers.

[jbkalmbach]

This is really cool, thanks for adding this in!

Can you please update the doc/uml/taskClass.uml file and the developer guide page?

Some pair tables and butler questions (these might kind of overlap but I'm just writing them down as they occurred to me reading the code):

• How do you envision pair tables being created/stored in production?
• Do we want to ingest pair tables for every run into their own collection or would we have a general pair table that is regularly updated and overwritten? Are we then going to have lots of collections with just a pair table as their contents?
• How we will know which pair tables to use when?

[jbkalmbach]

Do we want to allow the same intra exposure to be paired with multiple extra exposures? Would it be better to create a copy of the intra table and remove rows as they are paired?

[jmeyers314]

That's an interesting idea. I'm on the fence as to whether that should be the only mode of operation, or if this should be a config setting. Do you have a preference?

[jbkalmbach]

Yeah, I'm all for keeping our options open and using a config to set this.

[jmeyers314]

Responses to general questions:

How do you envision pair tables being created/stored in production?

We'll have to experiment. It might turn out that we can mostly rely on the automatic pairer. One option we could think about is creating a script that runs the automatic pairer and writes the result to a CSV file that we can then go and tweak by hand. I'm happy to do that here if you like, or leave it for a future ticket.

Do we want to ingest pair tables for every run into their own collection or would we have a general pair table that is regularly updated and overwritten?

Yeah, that's a good question. One pair table per collection is probably the easiest right now; I don't think butler products generally work very well with edit-in-place in mind, though you can mimic that with chain-collections where newer runs shadow older runs.

Are we then going to have lots of collections with just a pair table as their contents?

Possibly many run-collections. Hopefully fewer chain-collections though.

How we will know which pair tables to use when?

I think maintaining a default pair table chain-collection may make sense here. Quick estimate might be ~500 pairs per night and ~100 nights of commissioning. I don't think having a 50,000 row pair table is that bad.

Off-main pairing experiments will have to maintain their own collections. (or develop other Pairer subtasks).

[jmeyers314]

Hey Bryce,
I think I've addressed all your comments. Do you want to take a quick peak again? I also added a "generatePairTable.py" script to help assemble a table that can then be edited manually when using the TablePairer. Thanks!

[jbkalmbach]

Thanks for the generate script, all my comments have been addressed. Just one comment on the UML that I think is needed based upon what we've added before.

[jbkalmbach]

I think CutOutDonutsScienceSensorTaskConfig *-- ExposurePairer needs to be added as well.