Issue with nan in catalogues from deep stack #687

wfastrononomer · 2025-01-19T12:11:45Z

2025-01-15 14:25:13.86: exnumeric -tables /ibi/tmp/scos_2013062.0002/schema_17

0.dat -files /ibi/tmp/scos_2013062.0002/filelist_170_0.dat -cuEvent 506442 -csvp
ath /ibi/samba/share -sqlpath /ibi/scos/VDFS/trunk/sql -csvprefix cu04id506442_
ibi-1-202400_VSAVVV15_wfau -illumname /ibi/tmp/scos_2013062.0002/illumTables.dat
-outtables vvvDetectionRaw202400 -newobjid -1 -archive VSA

2025-01-15 14:25:21.68: vvvDetectionRaw202400 aperFlux2 25 16525 : -na

n : NaN value found:deepstackcat

What is causing this.

Other oddity, which may or may not be related. This is from processing these files:

/disk67/vsa/products/stacks/20241114_v2/e20241114_00170007310_dp_st.fit
/disk67/vsa/products/stacks/20241114_v2/e20241114_00170007310_dp_st_cat.fits
/disk68/vsa/products/stacks/20241204_v2/e20241204_00170007310_dp_st.fit
/disk68/vsa/products/stacks/20241204_v2/e20241204_00170007310_dp_st_cat.fits

But productID=7310 is in twice. Once processed on 20241114 and once on 20241204. Why? See issue #688

wfastrononomer · 2025-01-21T16:51:14Z

2025-01-15 14:25:21.68: vvvDetectionRaw202400 aperFlux2 25 16525 : -nan

: NaN value found:deepstackcat

Look in FITS catalogue and find how often. Find position on image and look at image. Is it weird values in image or conf?

wfastrononomer · 2025-01-21T17:01:44Z

for ii, hdu in enumerate(hdulist):
... if ii>0:
... tbdata = hdu.data
... mask = np.isnan(tbdata.field('Aper_flux_2'))
... if np.nonzero(mask)[0].size>0:
... print(ii, np.nonzero(mask)[0].size)
...
(2, 7)

data.field('X_coordinate')[mask]
array([1665.9089, 1923.2717, 1745.8866, 1546.486 , 983.702 , 1081.1755,
927.9202], dtype=float32)

data.field('Y_coordinate')[mask]
array([2012.9124, 2012.3914, 2012.6752, 2012.6948, 2012.5872, 2012.7638,
2012.9022], dtype=float32)

wfastrononomer · 2025-01-21T17:24:34Z

Struggling to use Gaia or ds9. ds9 loads up an image but in the past I could open other extensions, but clicking on File doesn't work.
Gaia gives /star/bin/gaia/gaia_wish: /teti/scos/usr/local/lib64/libgfortran.so.5: version GFORTRAN_10' not found (required by /star/lib/libprm.so.0) on /teti and /disk91/star-2023A/bin/gaia/gaia_wish: /unas/scos/usr/local/lib64/libgfortran.so.5: version GFORTRAN_10' not found (required by /disk91/star-2023A/lib/libprm.so.0 on /unas.

wfastrononomer · 2025-01-21T17:27:30Z

Tried looking at image/conf data values around 1665, 2012 on extNum=3. All values seem reasonable - min/max/isnan checks? Tried reversing x/y. Have I looked at the wrong extension. I checked out to 15 pixels either side. Need to do this more systematically.

tms-epcc · 2025-01-24T16:03:00Z

24/JAN/25

found some NaNs but hard to visualise these.
@esutorius says this is likely fixed but need to check with GAIA image visualisation

wfastrononomer · 2025-01-26T17:34:05Z

Got Gaia to work - ( setenv LD_LIBRARY_PATH '' && gaia /disk67/vsa/products/stacks/20241114_v2/e20241114_00170007310_dp_st_conf.fit ) &

So far can't see anything. Need to check more systematically.

Might be worth running imcore again more verbosely to see the issues. Check headers too...

wfastrononomer · 2025-01-27T10:34:18Z

Looking at extNum=3, the image and confidence.

data.field('X_coordinate')[mask]
array([1665.9089, 1923.2717, 1745.8866, 1546.486 , 983.702 , 1081.1755,
927.9202], dtype=float32)

data.field('Y_coordinate')[mask]
array([2012.9124, 2012.3914, 2012.6752, 2012.6948, 2012.5872, 2012.7638,
2012.9022], dtype=float32)

I have looked at the first 5 objects and their confidence. All the pixels around have reasonable values - no confidence of 0, no NaN in the arrays or extreme values. The first object has a weird almost triangular blob on the confidence image that might give odd values, although I would be very surprised if it gave NaN. The others all look fine. The only strange thing is that all have y=2012.

I will look at the headers.

Aper_flux_1 are all fine. Aper_flux_3 and Aper_flux_8 are all nan. The sequence numbers are 8197-8200, 8202, 8203, 8209.

wfastrononomer · 2025-01-27T15:27:24Z

When I ran it imcore again in verbose mode, extNum=3 had some strange estimated background values:

Computing background....

Background values:
-1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0
-1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0
-1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0 -1000.0
-1000.0 -1000.0 6707.6 6717.6 6716.5 6716.7 6716.4 6718.3 6717.2 6718.0
6716.6 6717.3 6717.0 6716.5 6716.3 6717.6 6719.5 6716.3 6716.7 6717.3
6717.3 6716.2 6716.6 6716.8 6716.1 6716.4 6718.1 6716.6 6716.3 6714.8
6717.2 6717.2 6716.7 6717.0 6718.2 6713.2 6717.5 6718.2 6718.2 6717.9
.....
6716.3 6715.5 6715.2 6716.5 6714.7 6716.4 6715.3 6716.2 6716.4 6717.0
6716.9 6720.1 6719.8 6716.5 6716.2 6716.9 6715.7 6716.8 6716.3 -1000.0
6716.8 6715.6 6717.0 6715.5 6714.5 6714.5 6715.5 6714.9 6715.5 6714.4
6714.6 6714.8 6715.4 6714.0 6714.2 6714.2 6713.8 6712.0 6714.3 6713.4
6714.6 6714.4 6716.5 6715.4 6716.2 6716.2 6715.0 6715.4 6715.4 6715.3
6717.6 6716.7

Filtered background values:
149805.8297372.2444938.6592505.2740072.1887638.81035205.21182771.61330338.114779
04.61625470.91773037.61920604.42068171.12215737.52363304.02510870.82658437.22806
003.82953570.53101137.03248703.83396270.23543837.03691403.23838969.83986536.5413
4103.04281670.04429236.54576803.53508063.01628452.0 60375.5115712.8171050.222638
7.8281725.6337063.2392400.5447737.8503075.2558412.6613749.8669087.2724425.077976
2.6835099.8890437.2945774.91001112.21056449.51111787.01167124.21222461.61277799.
11333136.81388474.11443811.21499148.61554486.51609824.21665161.81720499.41319721
.5614866.8 6717.5 6717.5 6717.6 6717.7 6717.9 6717.9 6717.8 6717.6 6717
.7 6717.6 6717.4 6717.3 6717.5 6717.5 6717.4 6717.4 6717.5 6717.4 6717
.2 6717.0 6716.8 6716.5 6716.5 6716.6 6716.5 6716.3 6716.3 6716.6 6716
.8 6716.8 6716.8 6716.6 6716.2 6717.3 6717.6 6717.8 6718.0 6718.0 6717
.8 6717.7 6717.7 6717.8 6717.8 6717.6 6717.6 6717.6 6717.5 6717.4 6717
.4 6717.5 6717.5 6717.4 6717.0 6716.5 6716.2 6716.3 6716.5 6716.4 6716
.2 6715.9 6715.9 6716.1 6716.2 6716.1 6716.1 6716.0 6717.1 6717.5 6717
.9 6718.0 6718.0 6717.7 6717.5 6717.6 6717.7 6717.7 6717.6 6717.4 6717
.4 6717.3 6717.2 6717.3 6717.5 6717.6 6717.3 6716.9 6716.3 6716.0 6716
.2 6716.5 6716.4 6715.8 6714.9 6714.5 6714.9 6715.3 6715.6 6715.6 6715
.5 6717.1 6717.5 6717.9 6718.0 6718.0 6717.7 6717.5 6717.6 6717.6 6717
.7 6717.5 6717.1 6716.9 6717.1 6717.2 6717.3 6717.5 6717.5 6717.2 6716
.9 6716.5 6716.1 6716.3 6716.4 6716.4 6715.7 6714.0 6713.0 6713.6 6714
.7 6715.3 6715.5 6715.6 6717.0 6717.5 6718.0 6718.2 6718.0 6717.9 6717
.9 6717.8 6717.7 6717.6 6717.4 6717.2 6716.9 6716.6 6716.8 6717.1 6717
.3 6717.3 6717.1 6717.0 6716.8 6716.5 6716.6 6716.7 6716.7 6716.0 6714
.3 6713.1 6713.9 6715.0 6715.6 6715.9 6716.2 6716.4 6717.2 6717.9 6718
.3 6718.3 6718.2 6718.2 6718.0 6717.9 6717.6 6717.3 6717.3 6716.8 6715
....
.2 6716.1 6716.2 6716.4 6716.4 6715.9 6715.8 6715.6 6715.2 6714.9 6714
.4 6714.3 6714.4 6714.5 6714.4 6714.1 6714.4 6715.1 6715.3 6714.9 6714
.5 6714.4 6713.9 6713.6 6713.9 6714.3 6714.7 6715.2 6715.8 6716.2 6716
.4 6716.2 6715.7 6715.6 6715.6 6715.8 6716.4 6716.7

Computing background statistics....

Sky level = 6716.70
Noise level = 10.70
Threshold = 13.37
Sat clips = 36746.1 36746.1
Extracting images....
Estimated FWHM (pixels) = 1.9
No. of objects used = 2144

wfastrononomer · 2025-01-29T17:22:25Z

In imcore_background.c:

if(nnp[j] > 0.25nbsizenbsize){
shist = hist[j];
imcore_medsig(shist,MAXHIST,MINHISTVAL-1,nnp[j],&skymed,&sigma);

            /* do an iterative 3-sigma upper clip to give a more robust
	   estimator */

	iclip = MAXHISTVAL;
	mcpix = nnp[j];
	skymedc = skymed;
	sigmac = sigma;
	for(iloop = 0; iloop < 3; iloop++) {
	    irej = 0;
            for(i = NINT(skymedc+3.0*sigmac); i <= iclip; i++)
                    irej += shist[i-MINHISTVAL];
	    if (irej == 0)
		break;
	    iclip = NINT(skymedc+3.0*sigmac) - 1;
	    mcpix = mcpix - irej;
	    imcore_medsig(shist,MAXHIST,MINHISTVAL-1,mcpix,&skymedc,
			  &sigmac);
	}
	bvals[l][j] = skymedc;
    } else {
	bvals[l][j] = -1000.0;
    }
}

I think that the background estimator fails because nnp[j] <= 0.25nbsizenbsize. Why?
Then reverts to the default.

I think that nnp is number of useful background measurements and it must be> a quarter of the local background area.

wfastrononomer · 2025-01-29T17:22:57Z

Check code and email Jonathon/Mike Irwin?

wfastrononomer · 2025-01-31T13:16:15Z

/unas/njc/usr/local/wsa/casutools/bin/imcore e20241114_00170007310_dp_st.fit e20241114_00170007310_dp_st_conf.fit test_cat3.fits 4 1.25 --crowd 1 --rcore 3 --nbsize 64 --noell --filtfwhm 2 --verbose --cattype 6

Seems that the issue is mflag == MF_ZEROCONF in thousands of pixels in each background square

Confidence image values do go low, but always above 0. Is there a minimum level?
Need to carefully check what happens. Is there a bug in the code, or some oddity I am missing. Which x and y positions exactly!

tms-epcc · 2025-03-05T15:23:35Z

05/MAR/25

no progress - only on one file

wfastrononomer added this to VVVX DR6 Jan 19, 2025

wfastrononomer converted this from a draft issue Jan 19, 2025

wfastrononomer mentioned this issue Jan 19, 2025

Multiple copies of some deep stacks produced. #688

Open

wfastrononomer moved this from To do to In progress in VVVX DR6 Jan 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with nan in catalogues from deep stack #687

Issue with nan in catalogues from deep stack #687

wfastrononomer commented Jan 19, 2025 •

edited

Loading

wfastrononomer commented Jan 21, 2025

wfastrononomer commented Jan 21, 2025 •

edited

Loading

wfastrononomer commented Jan 21, 2025

wfastrononomer commented Jan 21, 2025

tms-epcc commented Jan 24, 2025

wfastrononomer commented Jan 26, 2025

wfastrononomer commented Jan 27, 2025 •

edited

Loading

wfastrononomer commented Jan 27, 2025

wfastrononomer commented Jan 29, 2025

wfastrononomer commented Jan 29, 2025

wfastrononomer commented Jan 31, 2025

tms-epcc commented Mar 5, 2025

Issue with nan in catalogues from deep stack #687

Issue with nan in catalogues from deep stack #687

Comments

wfastrononomer commented Jan 19, 2025 • edited Loading

2025-01-15 14:25:13.86: exnumeric -tables /ibi/tmp/scos_2013062.0002/schema_17

2025-01-15 14:25:21.68: vvvDetectionRaw202400 aperFlux2 25 16525 : -na

wfastrononomer commented Jan 21, 2025

2025-01-15 14:25:21.68: vvvDetectionRaw202400 aperFlux2 25 16525 : -nan

wfastrononomer commented Jan 21, 2025 • edited Loading

wfastrononomer commented Jan 21, 2025

wfastrononomer commented Jan 21, 2025

tms-epcc commented Jan 24, 2025

wfastrononomer commented Jan 26, 2025

wfastrononomer commented Jan 27, 2025 • edited Loading

wfastrononomer commented Jan 27, 2025

wfastrononomer commented Jan 29, 2025

wfastrononomer commented Jan 29, 2025

wfastrononomer commented Jan 31, 2025

tms-epcc commented Mar 5, 2025

wfastrononomer commented Jan 19, 2025 •

edited

Loading

wfastrononomer commented Jan 21, 2025 •

edited

Loading

wfastrononomer commented Jan 27, 2025 •

edited

Loading