Creating Synthetic Exposure/FITS file by adding successive exposures

[gaitskell]

We have been looking in detail at the photometry from our Oculus Pro Camera (Starlight Xpress) with 150 deg FOV.
For exposure times greater than 1 sec the light entering the camera from an apparent magnitude 0 star (Arcturus, Vega) will cause the brightest pixel matched to the star to enter a non-linear regime. The non-linear regime starts for a pixel at a level above 70% of max pixel reading. Artifacts such as an oscillation of levels starts to occur between successive pixel exposures when pixels enter the non-linear regime so we want to stay away from it.

Is it possible to create a mode in indi-allsky that sums the raw images from rapid successive camera exposures and simply adds them creating a synthetic exposure. The summed result is saved as the synthetic FITS file. (The process would not save the separate input exposures as FITS files, just the final summed synthetic FITS. The synthetic exposure time would be the sum of the exposure times. The other FITS headers would match those appropriate to the first image in the sequence.)

https://github.com/aaronwmorris/indi-allsky/wiki/Image-order-of-operations

An example of how we would use it: Set a max exposure to 1 sec, and a synthetic image count of say 32. The initial camera process would take 32 images in succession, adding them together to create the synthetic image and synthetic FITS file (This would be in step 2 of the Order of Operations). The exposure time is the sum of the individual exposure times. The synthetic FITS would need to have a larger value range for each pixel. For example, use an unsigned integer 32 rather than original format for camera pixeks (unsigned integer 16) given that the summed image would likely overflow the 16 bits per pixel.

We want to do the summing at the early stage of the order of operations and work with the sythetic summed FITS to reduce the bandwidth required to transfer a large number of separate shorter exposure FITS files.

An alternative approach would be to insert an extra option in the pipeline for the FITS file handling, this occurs after the initial FITS files have been generated. When "n" initial FITS files have been generated a separate process would combined them into a new synthetic FITS file similar to that discussed above. The initial FITS files would be deleted and replaced by the single synthetic (summed) FITS. The synthetic FITS would then be the one that file handling stores, and copies to the remote site as it does currently. This approach has the simplicity of simply adding one new process in the file management queue. It does of course now require a temporary holding directory in which initial FITS files are accumulated prior to summing together.

The goal of summing images from the CCD is to improve the effective dynamic range of the CCD and to avoid the non-linear response regime for each individual exposure.

I realize the above options may not be consistent with your pipeline design. If you can see a way to insert such an option that would be great. Completely understand if it is too complex. I am aware of the existing stack option, but that is only applied to the later enhanced photo images?

As an aside, do you know how accurate the exposure times are? I assume the exposure time is sent to the specific camera hardware, and it determines how long to actually open the CCD wells?

[aaronwmorris]

I have been thinking about this. The stacking module would be an obvious place to do this. It is already designed to store a sequence of images (in memory) and perform an action on them. Performing a sum would not be very difficult. I do not know if it is necessarily appropriate, but I am still considering it.

As for the timing of the exposures... that can be a mixed bag. Most of the INDI servers are pretty good, but I have had one or two that gave problems. For instance, with my SVBony SV305, I had issues with about 33% of the images taking double the exposure time to complete. The exposure appeared to be correct, but for some reason, sometimes a 15s exposure would take 30s to complete. There are also times that some exposures arrive in less time than the requested exposure. This is the reason I started tracking the time it takes to receive exposures in the Lag view.

The exposure issue is why I shifted the capture process to a non-blocking loop early in development. Instead of waiting 15s after requesting a 15s exposure, indi-allsky immediately starts continuously polling the exposure control to check the status of the camera. (It also gives the opportunity to perform other tasks while waiting on exposures).

There is one other thing to consider in this case. Downloading the image from the camera will slow down the process considerably. You cannot start an exposure until the previous exposure is complete and downloaded. Downloading the exposure requires time, and it can range from 0.5s to 3-4s depending on the camera and size of the frame. If it takes 2s to download an image, it would require 90s to take 30 1s exposures.

A video streaming mode would negate the download time, but that is out of scope for indi-allsky.

[gaitskell]

Thanks for the notes. Agreed - the stacking module can handle it, just that the FITS stacking occurs right at the start of the flow. It may complicate the autoexposure feedback in your indi-allsky logic? (May be best to set a fixed expsoure time when in FITs stacking mode to make logic/control easier?)

(We are using an SX Oculus Pro - ICX267AL Sony SuperHAD.)
The CCD download overhead will be unavoidable if we are to maintain linearity, I will check the logs on our system to see what time that is consuming for 2.9 MB image download - I assume it is reported in the standard log file. When originally researching the camera I had a note that it takes 0.6 secs to download a frame over USB 2.0. This would be quite acceptable for multiple exposures.

There is a significant gain to be made in bandwidth/storage consumed for FITS transfer/handling as discussed with stacking at capture time. We would be stacking 32+ CCD images into one FITS.

We checked photometery for different exposure times and the light readings do appear to scale correctly with changes in exposure time (tested to within 5% error between different exposure times). If there are any "hung" exposures in our series they are much less than 1 in 5000 exposures (we don't see any evidence in data so I estimate that limit).

[gaitskell]

A recent log (see below) reports MainThread capture.saferun() [349]: Exposure received in 1.9148 s (0.9148)

I assume this implies ~0.9 sec download after the 1 sec exposure. This overhead is fine for each exposure.

(Or does it mean 1.9 sec on top of 1.0 sec exposure? Still fine.)

**admin@PiAllSky:/var/log/indi-allsky $ grep -m20 -C2 capture  indi-allsky.log**

Jun  5 00:00:17 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [349]: Exposure received in 1.9148 s (0.9148)
Jun  5 00:00:17 PiAllSky [INFO] Image-17-34895/MainThread processing.add() [489]: Image bits: 16, cfa: None
Jun  5 00:00:17 PiAllSky [INFO] Image-17-34895/MainThread processing._detectBitDepth() [575]: Image max value: 21517
--
Jun  5 00:00:17 PiAllSky [INFO] Image-17-34895/MainThread stretch.mode1_adjustImageLevels() [131]: Image levels in 0.0101 s
Jun  5 00:00:17 PiAllSky [INFO] Image-17-34895/MainThread processing.contrast_clahe_16bit() [1354]: Performing 16-bit CLAHE contrast enhance
Jun  5 00:00:17 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [214]: Camera last ready: 0.3s
Jun  5 00:00:17 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [215]: Exposure state: OK
Jun  5 00:00:17 PiAllSky [INFO] Capture-5-17321/MainThread capture.detectNight() [1203]: Sun altitude: -24:53:02.1
Jun  5 00:00:17 PiAllSky [INFO] Capture-5-17321/MainThread capture.detectMoonMode() [1221]: Moon altitude: -25:41:22.3, phase 2.5%
Jun  5 00:00:17 PiAllSky [INFO] Image-17-34895/MainThread processing._convert_16bit_to_8bit() [1040]: Resampling image from 16 to 8 bits
Jun  5 00:00:17 PiAllSky [WARNING] Image-17-34895/MainThread image.recalculate_exposure() [1609]: New calculated exposure: 1.00000000
--
Jun  5 00:00:17 PiAllSky [INFO] Image-17-34895/MainThread miscUpload.s3_upload_asset() [392]: Uploading to S3 bucket
Jun  5 00:00:17 PiAllSky [INFO] Image-17-34895/MainThread miscUpload.s3_upload_asset() [392]: Uploading to S3 bucket
Jun  5 00:00:20 PiAllSky [INFO] Capture-5-17321/MainThread capture.shoot() [1409]: Taking 1.00000000 s exposure (gain -1)
Jun  5 00:00:20 PiAllSky [INFO] Capture-5-17321/MainThread indi.set_number() [1339]: Setting Duration (s) = 1.0
Jun  5 00:00:20 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [425]: Image queue depth: 0
Jun  5 00:00:20 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [456]: Total time since last exposure 5.0400 s
Jun  5 00:00:22 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [349]: Exposure received in 1.9627 s (0.9627)
Jun  5 00:00:22 PiAllSky [INFO] Image-17-34895/MainThread processing.add() [489]: Image bits: 16, cfa: None
Jun  5 00:00:22 PiAllSky [INFO] Image-17-34895/MainThread processing._detectBitDepth() [575]: Image max value: 21879
--
Jun  5 00:00:22 PiAllSky [INFO] Image-17-34895/MainThread miscUpload.s3_upload_asset() [392]: Uploading to S3 bucket
Jun  5 00:00:22 PiAllSky [INFO] Image-17-34895/MainThread miscUpload.s3_upload_asset() [392]: Uploading to S3 bucket
Jun  5 00:00:25 PiAllSky [INFO] Capture-5-17321/MainThread capture.shoot() [1409]: Taking 1.00000000 s exposure (gain -1)
Jun  5 00:00:25 PiAllSky [INFO] Capture-5-17321/MainThread indi.set_number() [1339]: Setting Duration (s) = 1.0
Jun  5 00:00:25 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [425]: Image queue depth: 0
Jun  5 00:00:25 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [456]: Total time since last exposure 5.0345 s
Jun  5 00:00:27 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [349]: Exposure received in 1.9648 s (0.9648)
Jun  5 00:00:27 PiAllSky [INFO] Image-17-34895/MainThread processing.add() [489]: Image bits: 16, cfa: None
Jun  5 00:00:27 PiAllSky [INFO] Image-17-34895/MainThread processing._detectBitDepth() [575]: Image max value: 19727
--
Jun  5 00:00:27 PiAllSky [INFO] Image-17-34895/MainThread miscUpload.s3_upload_asset() [392]: Uploading to S3 bucket
Jun  5 00:00:27 PiAllSky [INFO] Image-17-34895/MainThread miscUpload.s3_upload_asset() [392]: Uploading to S3 bucket
Jun  5 00:00:28 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [214]: Camera last ready: 1.5s
Jun  5 00:00:28 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [215]: Exposure state: OK
Jun  5 00:00:28 PiAllSky [INFO] Capture-5-17321/MainThread capture.detectNight() [1203]: Sun altitude: -24:53:26.4
Jun  5 00:00:28 PiAllSky [INFO] Capture-5-17321/MainThread capture.detectMoonMode() [1221]: Moon altitude: -25:41:01.8, phase 2.5%
Jun  5 00:00:28 PiAllSky [WARNING] Capture-5-17321/MainThread indi.getCcdTemperature() [878]: Sensor temperature not supported
Jun  5 00:00:28 PiAllSky [INFO] Capture-5-17321/MainThread indi.getTelescopeRaDec() [868]: Telescope Coord: RA 16.19, Dec 41.74
Jun  5 00:00:30 PiAllSky [INFO] Capture-5-17321/MainThread capture.shoot() [1409]: Taking 1.00000000 s exposure (gain -1)
Jun  5 00:00:30 PiAllSky [INFO] Capture-5-17321/MainThread indi.set_number() [1339]: Setting Duration (s) = 1.0
Jun  5 00:00:30 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [425]: Image queue depth: 0
Jun  5 00:00:30 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [456]: Total time since last exposure 5.0380 s
Jun  5 00:00:39 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [349]: Exposure received in 9.0091 s (8.0091)
Jun  5 00:00:39 PiAllSky [INFO] Capture-5-17321/MainThread capture.shoot() [1409]: Taking 1.00000000 s exposure (gain -1)

[aaronwmorris]

Looks good. However, you can see one of the last exposures in your log there shows a 1s exposure received in 9s. There is no explanation why this happens, but it does. The timing being exactly 8s difference is very suspicious.

Jun  5 00:00:30 PiAllSky [INFO] Capture-5-17321/MainThread indi.set_number() [1339]: Setting Duration (s) = 1.0
...
Jun  5 00:00:39 PiAllSky [INFO] Capture-5-17321/MainThread capture.saferun() [349]: Exposure received in 9.0091 s (8.0091)

[gaitskell]

@aaronwmorris Trust all is well. Have you been able to see how to make the stacking of Synthetic FITS file possible at the front end? Is there a good way to implement this? Thanks, Rick

[aaronwmorris]

I am still thinking about this. I had another thought that this could be a new "camera" type. For instance, if you want a 30s exposure, it would take 30 1s exposures and return a single FITS frame with float32 data.

The challenge is the exposure feedback which adjusts the exposure based on current sky conditions. Most people use indi-allsky for the pretty pictures, not necessarily objective scientific data. :-)

I am still batting around ideas.

[gaitskell]

I think that anyone using front end stacking to create synthetic FITS would be happy with a specified exposure time - specifically you don't want the CCD exposure times changing based on exposure feedback since there are so many different ways to define an optimium exposure. Setting a fixed time per exposure works well in most scenarios.

[aaronwmorris]

I am still looking at this. I made a first attempt at creating a new camera driver and realized it would be more complex than I initially realized.

indi-allsky was designed around INDI (duh), which for the most part functions asynchronously. When configuration is changed, you have to check the status of the control to know when the new config is complete (like slewing a mount or taking an exposure).

The challenge I ran into is if I have to abstract the exposure controls, I have to do so asynchronously. This likely means I will have to create a second indi client connection in a thread to manage the exposures, separate from the main camera connection.

[gaitskell]

As an alternative one could create a seperate process that goes through the available local FITS files and generates combined synthetic files (by adding them) spanning some preset period of time. This would be independent (and after) the main capture and processing chain.

The upload/transfer processes to cloud would be configured to only upload *_synth.fits rather than all the local .fits files. There could be the option to remove the old .fits files after creating _synth.fits.

The logic for the process creating the _synth.fit could use simple time windows of a fixed time duration - say all fit files in successive 60 sec intervals (user selectable). The _synth.fit would only be generated after that time window has passed. I assume this could be an operation that would work using the internal database to keep a record of files created and destroyed?

From a process point of view this is may be more simple to implement? It would address the network bandwidth issue when exposures are set short(to avoid saturation in camera) at a rapid cadence. It reduces the large dataflow from local machine to servers.

[aaronwmorris]

Okay, I have made some progress on this. I have developed a new camera type which will be the "INDI Accumulator". Basically, the synthetic exposures will be summed prior to any processing and indi-allsky basically receives a single 32-bit unsigned FITS file with the synthetic result.

For the final image, the results are expected to roughly align with uint16 data. Any values greater than 65535 are cut off to 65535 and the image is processed as if the FITS were a normal uint16. However, the saved FITS file will have the original uint32 data.

The dark frame processing also supports using the synthetic data to remove dark current, etc, but this is optional.

This is currently in my dev branch and I am going to let this run for a few days on my systems to make sure there are no major issues.

[aaronwmorris]

Merged #1456 with the new Accumulator camera.