Cubical and Quartical comparision

[Kincaidr]

I am currently doing this comparison and need to confirm some things.

1.) Output products.

On the Quarti support group space on hangouts, a discussion between the output quartical products were made. Ian and John agreed on:

CORRECTED_DATA (output from output.columns: [corrected_data]) should contain K-corrected visibilities that correspond to the original sky, except for the DIR1 source which has been dE calibrated and subtracted. This is for the case where you might want to proceed onto addition DDE calibration steps with extra rounds of deconvolution to get the benefit of having the problem source removed.

CORRECTED_RESIDUALS should contain visibilities that are K corrected, and have dE-corrected DIR1 removed, but additionally the entire sky model is subtracted from the visibilities. This is for cases when we just want to subtract the all the original sources as cleanly as possible, for example to rephase and image the Sun. No additional deconvolution takes place for the original field centre, and we do not need an extra step that subtracts the field sources from the data.

output.products: [corrected_data, corrected_residual] in quartical corresponds to out_mode: sc and out_mode: sr in Cubical. I want to ask if the above explanation of output products also apply to Cubical? I've never set this out_mode parameter when using Cubical for DDE calibration before and therefore it was always on out_mode: sc by default. Therefore, if it is the same as the above, then I was never subtracting my sources?

2.) Solver types.

When combining terms in cubical i can only set them to complex like:

        sol_jones: [K,de]
        k_type: complex-diag
        de_type: complex-2x2

In quartical i have got this:

         solver.terms: [K,de]
         K.type: phase
         K.direction-dependent
         de.type: complex

How much of a difference would this make?

[JSKenyon]

output.products: [corrected_data, corrected_residual] in quartical corresponds to out_mode: sc and out_mode: sr in Cubical.

Yes, to the best of my knowledge. I did check the CubiCal code and it should be doing the same thing.

I want to ask if the above explanation of output products also apply to Cubical?

Yes, parameter names not withstanding.

I've never set this out_mode parameter when using Cubical for DDE calibration before and therefore it was always on out_mode: sc by default. Therefore, if it is the same as the above, then I was never subtracting my sources?

sc mode in CubiCal never subtracts anything, so that may be true. Has CubiCal never given you improvements in the past? This might be why.

How much of a difference would this make?

I am not sure I follow. The following should be the equivalent:

# CubiCal
sol_jones: [K,de]
k_type: complex-diag
de_type: complex-2x2
de_dd_term: True

# QuartiCal
solver.terms: [K,de]
K.type: diag_complex
de.type: complex
de.direction_dependent: true

[Kincaidr]

These are comparative results of quartical with cubical using (almost) the same settings:

I had to have --k-update-type phase-diag for Cubical to match k.type phase in Quartical since a dd solve in Cubical by default mean's both terms have to be complex-2x2.
As you can see, the over-subtraction is not as bad in cubical as what I was getting with Quartical. It gets better when I keep the phase term as complex ( --k-update-type full ):

If I want to replicate this RHS with Quartical I would need to set k.type complex; if I do I get this messs:

Cubical has a lot of additional options as compared to Quartical. I have set the main ones to be the same. This includes solvers, sovler types, solution intervals and mad-max settings. Anything else that could influence the results drastically that I missed?

[IanHeywood]

I think something is wrong with the imaging. These positive / negative artefacts with symmetry are common in most of the images with varying degrees of severity. They look like the ones that pop up when there is an insufficient number of w-planes, although I don't know what gridder you're using. These can easily end up in models when using masking/cleaning to construct them.

[JSKenyon]

Thanks for the input @IanHeywood! There are a number of weird things going on here - I think I have convinced myself that the QC and CC configs are sufficiently different that a comparison is not yet fair. @Kincaidr I am busy crafting a QC config file which is as close to identical to your best CC run as I can make it. Will get it to you today or tomorrow.

[Kincaidr]

I just used the standard clean options. I am not sure what you mean by your last sentence: "These can easily end up in models when using masking/cleaning to construct them." The model was constructed in the usual way with the predict step in wsclean.

[IanHeywood]

I guess the visibility model is predicted from the (spectral) clean components that wsclean produces as part of the deconvolution process. These artefacts are bright enough to end up in the cleaning masks and therefore the model, and so their presence will be reinforced by selfcal. I'm not saying that's the cause of your primary issue here but it's definitely something to be avoided. I'd consider switching to the w-gridder, I've been using that by default for a while now, it seems to do things better and faster than w-stacking in general.

[JSKenyon]

Thanks for doing these comparisons! Please send/post the following - your full configs for both CC and QC which produced the above, as well as the path to your data in an appropriate state (i.e. with flags correct up to the point that you know them).

…

On Mon, 7 Nov 2022, 15:41 Kincaidr, ***@***.***> wrote: These are comparative results of quartical with cubical using (almost) the same settings: [image: image] <https://user-images.githubusercontent.com/53697426/200320688-bad23f8c-2cce-419b-9127-4e3dee6bd67f.png> I had to have --k-update-type phase-diag for Cubical to match k.type phase in Quartical since a dd solve in Cubical by default mean's both terms have to be complex-2x2. As you can see, the over-subtraction is not as bad in cubical as what I was getting with Quartical. It gets better when I keep the phase term as complex ( --k-update-type full ): [image: image] <https://user-images.githubusercontent.com/53697426/200321461-3cb7e9af-7772-48af-b93d-8bf1b14be041.png> If I want to replicate this RHS with Quartical I would need to set k.type complex; if I do I get this messs: [image: image] <https://user-images.githubusercontent.com/53697426/200322404-da9c1a99-5e01-4a29-88d1-6bb6017874f9.png> Cubical has a lot of additional options as compared to Quartical. I have set the main ones to be the same. This includes solvers, sovler types, solution intervals and mad-max settings. Anything else that could influence the results drastically that I missed? — Reply to this email directly, view it on GitHub <#204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSHDWLD5TZFJZM37SCJFQTWHEBJHANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you commented.Message ID: ***@***.***>

[Kincaidr]

wsclean -name obs1lb/im5/im5 -data-column CORRECTED_RESIDUAL -channels-out 8 -weight briggs 0 -size 6076 6076 -scale 1.5asec -niter 2000000 -fits-mask obs1lb/im3/im3-mask.fits -mgain 0.9 -padding 1.3 -temp-dir temp_dir Abell3376_timechannel_t11_c4.ms

[JSKenyon]

Please do me a favour @Kincaidr - restore the best flags you have into the MS (I think you mentioned that it was roughly 40% flagged). Then run goquartical-backup Abell3376_timechannel_t11_c4.ms columns.bkp FLAG and goquartical-backup Abell3376_timechannel_t11_c4.ms columns.bkp FLAG_ROW. You will then need to point me to wherever you decide to store columns.bkp. This will ensure that I am using the same flags as you.

[Kincaidr]

ok done, columns.bkp file is stored here as before: /home/kincaid/xova_analyses_new_timechannel/xova_timechannel

[IanHeywood]

I noticed you're using the same MS for cubical and quartical. Are you resetting the model (as well as flags) between runs? If you do something like

• run cubical
• image
• run quartical
• image
• compare

then the quartical run will perhaps be starting from the model that results from the deconvolution of the cubical run, rather than the model that was there prior to the cubical run. If you're peeling bright sources then those two models will be very different. Sorry if I'm stating the obvious here, but it's worth checking.

[Kincaidr]

Yes actually this is a good point. I have been resetting the flags but not the model. Let me make copies of the MS, one for cubical run and one for quartical and re-run.

[JSKenyon]

Great! Note that the QC madmax may have been catastrophically overflagging (which could explain the mess you saw). As a rule, run goquartical-summary before and after runs just to check that flag percentages remain sensible. That would be my first suspect, but I can try and take a look this week.

…

On Mon, 7 Nov 2022, 15:56 Kincaidr, ***@***.***> wrote: Quartical: goquartical input_ms.path=Abell3376_timechannel_t11_c4.ms input_ms.data_column=DATA input_ms.weight_column=WEIGHT_SPECTRUM input_ms.time_chunk=128 input_ms.freq_chunk=0 input_ms.is_bda=False input_ms.group_by=['SCAN_NUMBER','FIELD_ID','DATA_DESC_ID'] input_ms.select_uv_range=[0.0,0.0] input_model.recipe=MODEL_DATA~DIR1_DATA~DIR2_DATA~DIR3_DATA:DIR1_DATA:DIR2_DATA:DIR3_DATA input_model.beam_l_axis=X input_model.beam_m_axis=Y input_model.invert_uvw=True input_model.source_chunks=500 input_model.apply_p_jones=True output.gain_directory=gains.qc output.log_directory=logs.qc output.overwrite=True output.products=['corrected_data','corrected_residual'] output.columns=['CORRECTED_DATA','CORRECTED_RESIDUAL'] output.flags=True output.apply_p_jones_inv=True output.subtract_directions=[1,2,3] mad_flags.enable=False mad_flags.threshold_bl=5.0 mad_flags.threshold_global=5.0 mad_flags.max_deviation=5.0 solver.terms=['K','de'] solver.iter_recipe=[100,50] solver.propagate_flags=True solver.robust=False solver.threads=1 solver.convergence_fraction=0.99 solver.convergence_criteria=1e-06 solver.reference_antenna=0 dask.workers=1 dask.scheduler=threads K.type=complex K.solve_per=antenna K.direction_dependent=False K.time_interval=8 K.freq_interval=0 K.interp_mode=reim K.interp_method=2dlinear K.respect_scan_boundaries=True K.initial_estimate=False de.type=complex de.solve_per=antenna de.direction_dependent=False de.time_interval=10 de.freq_interval=64 de.interp_mode=reim de.interp_method=2dlinear de.respect_scan_boundaries=True de.initial_estimate=True Cubical: gocubical cc-parsets/solve-kde.parset --data-ms Abell3376_timechannel_t11_c4.ms --data-time-chunk 10 --dist-max-chunks 4 --dist-ncpu 5 --dist-nthread 0 --data-freq-chunk 32 --model-list MODEL_DATA+-DIR1_DATA+-DIR2_DATA+-DIR3_DATA:DIR1_DATA:DIR2_DATA:DIR3_DATA --sol-jones K,de --sol-term-iters 100,50 --k-type complex-2x2 --k-time-int 4 --k-update-type phase-diag --k-freq-int 0 --out-column CORRECTED_RESIDUAL --out-mode sr --out-dir obs1lb/solve1-kde --out-subtract-dirs 1,2,3 --debug-pdb False --dist-safe 0 --dist-nworker 0 Cubical parset file also used: solve-kde.txt <https://github.com/ratt-ru/QuartiCal/files/9952212/solve-kde.txt> Ill reset the flags now. Data is on crosby at: /home/kincaid/xova_analyses_new_timechannel/xova_timechannel — Reply to this email directly, view it on GitHub <#204 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSHDWL25NFKBOMSLFE7I3DWHEC7VANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you commented.Message ID: ***@***.***>

[Kincaidr]

Ok let me check if this was the problem

[Kincaidr]

Don't think it was this. It went from 40% flagging to 60%

[bennahugo]

Mmm yea it is a bit trigger happy it seems, but it should not oversubtract like that. I wonder if it is not worth plotting the dE gain terms to see if there are an amplitude issue in the way the model is being applied, or if we should not write out a model out column and compare directly to the addition of models 1 through 3?

…

On Mon, 07 Nov 2022, 18:07 Kincaidr, ***@***.***> wrote: Don't think it was this. It went from 40% flagging to 60% — Reply to this email directly, view it on GitHub <#204 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6RS4AAABSQO2NOBKSLWHESMRANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[JSKenyon]

I am going to list my current findings here, which will hopefully help us figure out what is going on:

1. The flags which I asked @Kincaidr to backup are consistent with major over-flagging. This is the first and most important thing to check i.e. that the flags are both relatively good and that they are what you expect to be there.
2. The MODEL_DATA column is/was very strange at the point that I copied the data over. For the purposes of these experiments, I would advise against updating MODEL_DATA to ensure that the implementations have the same inputs.
3. The CC and QC configs were very different (and the de term in QC was not configured correctly). The choices of chunk size in CC would have effectively changed the solution intervals. I have crafted a QC config which should replicate the behaviour of the CC config. That said, having run it myself, I know that it does not produce good results due to points 1) and 2).
4. MAD flagging should not be enabled in either CC or QC until the very last stages of self-cal (i.e. with a very complete model) as it can and will result in overflagging. We want to eliminate possible causes of the above issues so leave this disabled.

I have attached a CC and a QC config below. The CC config should be consistent with the results you presented above, and I have crafted a QC config to match. Note that these configs can be further improved but I wanted to keep them as close to what you already had as possible. Hopefully this can help us track down the source of the problems.

cc-solve-kde.parset.txt
qc-solve-kde.yaml.txt

I cannot stress enough that the inputs into QC and CC need to be identical for us to be able to draw meaningful conclusions.

[Kincaidr]

1.) Could you please tell me how you concluded that there were over-flagged? According to the flagging summary of Caracal they were around 55%, which is normal for MeerKAT data.
2.) I'll use separate MS's for each run to solve this issue.
3.) Will use these configs.
4.) I'll disable this from the start.

[JSKenyon]

1.) Could you please tell me how you concluded that there were over-flagged? According to the flagging summary of Caracal they were around 55%, which is normal for MeerKAT data.

It is my opinion that looking at the global percentage of flagged data is an inadequate method of determining the quality of the flagging. Similarly, I dislike the fact that flagging percentages of 55%-60% are regarded as "normal" for MeerKAT; I attribute this to the fact that flagging is often neglected as a processing step. In this particular case, I used rfigui which is part of AOFlagger to inspect waterfall plots of the data. These plots indicated that many of the shorter spacings were overflagged, and that the flags were very "patchy". This could be attributed to a previous MAD flagging run which, as mentioned above, can lead to overflagging. These "patchy" flags can make calibration very difficult as there may be very little data in a given solution interval - I will think about what QC should do in this case. I will say that I don't think that the overflagging is the root cause of the problems; see below.

I have spent the whole day digging into this problem. At this point I am fairly confident that the model (the component in the MODEL_DATA column) is definitely incorrect in the data I copied. In particular, there are some sub-bands (particularly 0003) where the sub-band model image is absolutely awful. I guess I would suggest rerunning everything and carefully checking that the model images seem compatible with the data at all stages of the processing.

[Kincaidr]

Could you provide me with the rfigui command you used? I usually use surfvis for this.

[IanHeywood]

Regarding (2), you can also use wsclean's -no-update-model-required switch to ask it to leave the model column alone. Using this is addition to two Measurement Sets might be double insurance if you're running repeated tests.

[JSKenyon]

Yep - good point Ian, I was tripped up by this myself earlier today. To be clear @Kincaidr, wsclean will update the MODEL_DATA column based on the clean components by default. This means every time you image you will change the model. If you were trying out a few different calibration strategies in a row (and imaging the results), you would end up using a different model each time. Apologies that this is so tortuous.

[JSKenyon]

If you are on a Rhodes box you will need to SSH with -XC. Then rfigui path/to/ms. Then you can just play around in the gui.

…

On Fri, 11 Nov 2022, 18:04 Kincaidr, ***@***.***> wrote: Could you provide me with the rfigui command you used? I usually use surfvis for this. — Reply to this email directly, view it on GitHub <#204 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSHDWOTGFMHFEXTUEUWJP3WHZVAVANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you commented.Message ID: ***@***.***>

[bennahugo]

Ok I would suggest we look at the flagging in more detail as a second step - afterall the cubical images look really good, so if we can reproduce them with quartical it is good enough as a result already - any snr boost on top of that is a bonus. I agree the flagging is a bit heavy on the short spacings, but I'm not sure what was used (and what configurations applied) to flag the data.

…

On Fri, 11 Nov 2022, 18:07 JSKenyon, ***@***.***> wrote: If you are on a Rhodes box you will need to SSH with -XC. Then rfigui path/to/ms. Then you can just play around in the gui. On Fri, 11 Nov 2022, 18:04 Kincaidr, ***@***.***> wrote: > Could you provide me with the rfigui command you used? I usually use > surfvis for this. > > — > Reply to this email directly, view it on GitHub > < #204 (reply in thread) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ABSHDWOTGFMHFEXTUEUWJP3WHZVAVANCNFSM6AAAAAAQV4S3Q4 > > . > You are receiving this because you commented.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6VMFBLA56LZAADN6CDWHZVN5ANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you commented.Message ID: ***@***.***>

[JSKenyon]

You are absolutely correct @bennahugo - I agree that the goal is to have QC replicate or improve the CC result. I think the problem at present is that I do not know for certain that the model was the same for the QC run as it was for the CC run. When imaging MODEL_DATA (in the MS I originally copied), I noticed that some of the DD sources appeared to be negative in some sub-bands. This suggests that MODEL_DATA had already been replaced by the results of a post-peeling WSCLEAN run and would easily explain why the DD term in QC does a poor job (subtracting the DD columns from MODEL_DATA in which they are already peeled off will give large negative values). To be clear, I am not claiming that there cannot be a problem in QC (there absolutely can be). I would need a copy of the MS in which the MODEL_DATA column and the flags are known to be correct i.e. not the result of a previous calibration attempt. Also bear in mind that I can always be completely wrong - it has happened before. :P Happy to keep looking into this though.

[Kincaidr]

The original MS is here: /home/kincaid/xova_analyses_new_timechannel/xova_timechannel called Abell3376_timechannel_t11_c4.ms.

I have used rfigui to look at a few baselines, and I am not sure what to look for in terms of bad data from these plots.

You said shorter spacing were over-flagged and some flags were very patchy.
I have included a short spacing plot and a longer spacing plot:

So over-flagging would mean the pink regions are dominating, which I do see this trend when looking at the shorter spacing as compared to the longer ones as shown above. For the patchiness feature, how do we idetify this and what could it indicate?

[JSKenyon]

This looks a little better than the flags I ended up looking at - as mentioned, it could have been a prior MAD flagging step that was causing problems. Thanks for the original data - I will take a look to see if I see anything alarming. That said, there is no model in that MS so I cannot try the calibration. Can I ask you to please run your pipeline for QuartiCal and keep all intermediary images (and let me know there they are)? Use the new configs and see what happens. If something still goes wrong, I will need the model at the point that things went wrong, otherwise figuring out the problem will be almost impossible.

[Kincaidr]

Yes it is currently running now

[Kincaidr]

New results looking good, no over-subtraction now:

Last 2 images: Comparison of the new 3GC image with the older over-subtracted problematic one.

Will try this for bda now.

[Kincaidr]

1st image is actually (after 1GC)

[JSKenyon]

Those look much better @Kincaidr! I have actually been playing around with your recipe on my end, trying to understand why you needed to use phase in QC but f-slope in CC. Turns out the reason is #215. Setting input_model.apply_p_jones=false and output.apply_p_jones_inv=false will make QC delay and CC f-slope give pretty much indistinguishable results. Regardless, great work on your end - I am glad that things are beginning to make sense again!

[bennahugo]

Very good. Please note Ian's issue. You may need to up the number of w layers. Jon also pointed out ghost sources which is there in the 1gc maps. I think we may need to increase the padding factor to 2.0 in the final recipe

…

On Thu, 17 Nov 2022, 20:47 Kincaidr, ***@***.***> wrote: New results looking good, no over-subtraction now: [image: image] <https://user-images.githubusercontent.com/53697426/202531648-7a90d28e-6008-466c-bcb9-e138320a22bb.png> Last 2 images: Comparison of the new 3GC image with the older over-subtracted problematic one. Will try this for bda now. — Reply to this email directly, view it on GitHub <#204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6UOCFIHA5VSKPMBR5LWIZ4VNANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Kincaidr]

Yes I used the w-gridder for these imaging runs and padding: 1.3. Regarding the w-layer issue, I should then probably include nwlayers_factor. I can try see what nwlayers_factor: 12 does, what do you think?

[IanHeywood]

I've just been using the w-gridder for the last year or so. I haven't had the issue with the artefacts and haven't had to worry about setting the w-layers parameter.

[bennahugo]

Yup as discussed I think QC should bomb out if you have a multi dimensional model cube with no dEs - with a switch to plough on ahead if the user so desires. I didn't realise it was not solving for dE terms from the parameters.

…

On Fri, 18 Nov 2022, 08:28 JSKenyon, ***@***.***> wrote: Those look much better @Kincaidr <https://github.com/Kincaidr>! I have actually been playing around with your recipe on my end, trying to understand why you needed to use phase in QC but f-slope in CC. Turns out the reason is #215 <#215>. Setting input_model.apply_p_jones=false and output.apply_p_jones_inv=false will make QC delay and CC f-slope give pretty much indistinguishable results. Regardless, great work on your end - I am glad that things are beginning to make sense again! — Reply to this email directly, view it on GitHub <#204 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6RKDPQIBHEWK7WH4VTWI4OX5ANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[landmanbester]

Yes I used the w-gridder for these imaging runs and padding: 1.3. Regarding the w-layer issue, I should then probably include nwlayers_factor. I can try see what nwlayers_factor: 12 does, what do you think?

I don't think any of those parameters have any effect when using the wgridder in wsclean because it computes them based on the desired accuracy (which is hardcoded to 1e-4 if I am not mistaken)

[bennahugo]

Ah good point. You may need to swap back to the old gridder to get better accuracy if that tolerance can't be set externally

…

On Fri, 18 Nov 2022, 12:00 Landman Bester, ***@***.***> wrote: Yes I used the w-gridder for these imaging runs and padding: 1.3. Regarding the w-layer issue, I should then probably include nwlayers_factor. I can try see what nwlayers_factor: 12 does, what do you think? I don't think any of those parameters have any effect when using the wgridder in wsclean because it computes them based on the desired accuracy (which is hardcoded to 1e-4 if I am not mistaken) — Reply to this email directly, view it on GitHub <#204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6V3LISR25FX7NSCV2TWI5HVFANCNFSM6AAAAAAQV4S3Q4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[IanHeywood]

It can be set manually with the -wgridder-accuracy parameter.

[Kincaidr]

Ok i see this option with wsclean 3.1. How would I determine the -wgridder-accuracy?

[landmanbester]

I think it is set to 1e-4 by default. Should be good enough but you can always set it to 1e-5 and see if things improve. I think the maximum accuracy (for single precision) is 5e-6

[landmanbester]

It can be set manually with the -wgridder-accuracy parameter.

Good to know. Last I used wsclean it was hardcoded

[Kincaidr]

Cubical run now, results almost identical (see last 2 images):

Just regarding what you said earlier:

trying to understand why you needed to use phase in QC but f-slope in CC. Turns out the reason is #215. Setting input_model.apply_p_jones=false and output.apply_p_jones_inv=false will make QC delay and CC f-slope give pretty much indistinguishable results.

So yes, for the 2GC part. I used the f-slope solver and then switched to complex-2x2 with update type phase-only for 3GC. But if I did complex-2x2 with update type phase-only instead of f-slope for 2GC, it would probably make no difference.