BDA results- oversubtraction

[Kincaidr]

We are trying to solve the over-subtraction problem we are getting with both BDA and regular averaging.

For re-cap, the final K + de solve images looked like:

Regular time channel left, BDA right:

We suspected that the shorter spacings are picking up a lot of the low-level unmodelled flux that are biasing the solutions towards the sources and over-subtracting as a result. To check this, I did the K + de with 750 , 1000, 1450 UV-cuts. Below I have shown these dirty images along with the no-UV-cut dirty image. They are ordered as no uv-cut (top left), 750 (top right), 1000 (bot left), 1450 (bot right) for the two problematic sources :

Ive found the min for these two sources too see if over-subtraction results and possible trends. I got:
Source 1 (top):
no uvcut: -1.74; 750 uvcut: -1.90; 1000 uvcut: -1.58; 1400 uvcut: -1.75
Source 2 (bottom):
no uvcut: -1.94; 750 uvcut: -1.55; 1000 uvcut: -1.75; 1400 uvcut: -1.88

It seems like increasing the uv-cut is resulting in the same over-subtraction as no-uvcut. Is there any other way I can quantify this over-subtraction?

[JSKenyon]

Wait, are the first two images cleaned and the remaining images dirty? The dirty images don't look particularly oversubtracted to me (are they the correct images?). The large holes you are seeing seem much clearer in the deconvolved images but that means it is very difficult to tell where they are coming from.

Could you please include images of the unaveraged data as a point of comparison? Also which imager are you using? Could you please also post the model images or point me to the fits files - I would like to see the magnitude of the negative components near those sources.
Additionally, I am not sure how severe those UV cuts are. For some reason I feel like they may be quite high, but my practical experience with UV-cuts is limited.

How good is the model for the DDE affected sources? Apologies for the million questions.

[Kincaidr]

1.) Yes the first two images are deep cleaned images, and the UV-cutouts were done for the dirty images only to save time. Yes, they do not look as over-subtracted as the deep images (the deep images are about a magnitude more subtracted), but there are still holes there nevertheless, just not as deep as the cleaned ones. Is this not because they are only dirty images?

2.) I can do the un-averaged ones now. I was using WSClean with niter 0, so no model was created for the UV-cuts. Yes the UV-cuts are quite high, but this was just for experimental reasons to see if it could make a difference in preventing the over-subtraction.

3.) For the above-cleaned image, the model image with the corresponding cleaned image for the regular average is here:

Or you can access it over here /home/kincaid/xova_analyses_new_timechannel/xova_timechannel/obs1lb/im5 on crosby

[JSKenyon]

Great! I have had a dig through the images. What where your solution intervals on the direction dependent term?

[Kincaidr]

I used this:

    quartical_2:
      cab: quartical
      info: "Peel off axis source"
      params:
        input_ms.path: '{recipe.ms}'
        input_ms.is_bda: true
        input_ms.weight_column: WEIGHT_SPECTRUM
        input_ms.time_chunk: 0
        #input_ms.select_uv_range: [1400,0]
        solver.terms: [K,de]
        K.type: phase
        K.direction_dependent: false
        K.freq_interval: '0'
        K.time_interval: '4' 
        de.time_interval: '10'
        K.initial_estimate: false
        de.type: complex
        de.freq_interval: '64'
        solver.iter_recipe: [100,50]
        input_model.recipe: MODEL_DATA~DIR1_DATA~DIR2_DATA~DIR3_DATA:DIR1_DATA:DIR2_DATA:DIR3_DATA
        output.overwrite: 'true'
        output.products: [corrected_data, corrected_residual]
        output.columns: [CORRECTED_DATA,CORRECTED_RESIDUAL]
        output.subtract_directions: [1,2,3]

[JSKenyon]

You could try setting solver.iter_recipe=[100,100]. The dE term may not converge, so letting it iterate longer may slightly improve results. One other thing you can try (just as a point of comparison), it to make K.type=diag_complex. This is slightly dangerous if your model is incomplete but I would be interested to see if it improves things. If there is a residual time dependent amplitude error, the dE may not have sufficient resolution to capture it i.e. the source of the problem could be the absence of an amplitude DI term rather than the dE term itself. Again, I know that this is usually avoided due to concerns about flux suppression but I do think it is a good thing to check.

[Kincaidr]

One other thing you can try (just as a point of comparison), it to make K.type=diag_complex.

About this, slightly off-topic, I am also currently doing the quartical and cubical comparison , and in order for me to do this exact solve in cubical I would need to make the K.type term either complex-2x2 or robust-2x2. For a complete model, could this drastically change things?

[bennahugo]

Just a note: I think we should also clarify here that K is not delay "K" here - it is just a phase only gain if I understand K.type correctly right (as in "phase-diag" in cubical equivalent speak)? I can't immediately see what types of gains are available from the quartical/config/helpstrings.yaml helpstrings.

Still at a bit of puzzled, but yes it could be a global amplitude bias, quite correct. I think we can reasonably confidently do K + G(amp) + dE with the model we have.

[JSKenyon]

https://quartical.readthedocs.io/en/latest/gain_types.html - this is a little out of date, but you can get the gist of things there. You are correct that phase in QuartiCal is phase-diag in CubiCal (I renamed it as a 2x2 phase-only term doesn't make sense). You can of course experiment with a KGdE chain, but the simplest experiment would probably be GdE with G set as diag-complex with a similar resolution to the current K.

[JSKenyon]

You could even try setting G.type=complex to see if any of the remaining errors are due to leakage.

[Kincaidr]

BDA now consistent with regular averaging. Final DD-subtracted images:

[o-smirnov]

Yes! 🍾

[bennahugo]

Great work. I think we can start working on the paper with this in hand.

…

On Mon, 28 Nov 2022, 11:53 Oleg Smirnov, ***@***.***> wrote: Yes! 🍾 — Reply to this email directly, view it on GitHub <#202 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6R65JGXXU4DRBSLOB3WKR6KFANCNFSM6AAAAAAQSE2PD4> . You are receiving this because you commented.Message ID: ***@***.***>

[JSKenyon]

Apologies for not replying immediately @Kincaidr! Your results look awesome - thanks for all the work you have done on this.

[Kincaidr]

I am thinking of doing this for one more data-set, I already have it at hand. If it's too much trouble I can just leave it, I will start the draft paper simultaneously regardless. What do you think of showcasing this for 2 fields?

[bennahugo]

Hi Robert I think one field is enough to demonstrate (I don't want to unnecessarily draw out the paper). Remember that we have to recap the theory, the compression implementation qualification and quantification testing plus discuss the calibration routine implementation details and then the demonstration of calibration on real data along with a reasonable detailed discussion on the pipeline. Thoughts @sjperkins @JSKenyon ??

…

On Mon, Dec 5, 2022 at 11:31 AM Kincaidr ***@***.***> wrote: I am thinking of doing this for one more data-set, I already have it at hand. If it's too much trouble I can just leave it, I will start the draft paper simultaneously regardless. What do you think of showcasing this for 2 fields? — Reply to this email directly, view it on GitHub <#202 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6SNTSD7THXFB7KDBSTWLWY73ANCNFSM6AAAAAAQSE2PD4> . You are receiving this because you commented.Message ID: ***@***.***>

-- -- Benjamin Hugo PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[sjperkins]

Hi Robert I think one field is enough to demonstrate (I don't want to unnecessarily draw out the paper). Remember that we have to recap the theory, the compression implementation qualification and quantification testing plus discuss the calibration routine implementation details and then the demonstration of calibration on real data along with a reasonable detailed discussion on the pipeline. Thoughts @sjperkins @JSKenyon ??

This sounds sensible to me.

[JSKenyon]

That seems fair, although we can change out minds later if we feel the paper needs the additional results. Now that there is a sort of recipe for this I don't think running it on additional fields would be too arduous should the need arise.