append gains for each chunk instead of replacing them

[SpheMakh]

calico-stefcal replaces the pickle file that stores the gains each time it processes a new chunk. It would be better if it appended to the file instead.

[IanHeywood]

Relevant discussion:

#15

[IanHeywood]

For the record, the fix suggested in #15 where chunks of data are looped over with new names for stefcal_gain.table (and stefcal_diffgain.table) works. An example is given here, if you can bear my usual messy scripts:

https://github.com/IanHeywood/videocal/blob/master/per_scan_per_band_calibration.py

For VLA data I usually loop over SPWs and contiguous blocks of scan numbers (you probably don't want to solve over any breaks for visits to a calibrator anyway). There's a function for pulling these out of the MS in the above script.

Another catch is that the args for mqt.run don't like spaces, so you have to make use of the pokeTDLconf function if you want to pass different TaQL strings, as per the above script and this issue: ratt-ru/pyxis#47

Not sure how all this works if you're using stefcal.stefcal instead of mqt.run.

[SpheMakh]

This is great. I definitely shouldn't be solving over breaks for visits to calibrators. I'm adding some of the sweetness in your script to the stimela invocation of mqt.run for calibration.

[IanHeywood]

Sweet.

The reason the contiguous blocks of scan numbers are required is that VLA scans are limited to about 5 minutes. So if you're observing the same target for 20 minutes that continuous block of visibilities will actually contain 4 consecutive scans. This step probably isn't necessary for other telescopes (it's not required for ATCA or ASKAP, for example).

I've also got some scripts for plotting the gain pickles, but they're a bit clunky and not very general in that they assume the same gain table naming scheme I used in the calibration script to get the time and frequency tiling correct. The other thing is that the time labelling is given in units of the solution interval, as both the MS and the tdlconf would need to be consulted to turn this into actual time stamps.

I can tidy these up and upload them if you want.