Scaling of initial K0 guesses with TIFF files

[thomasmfish]

We are trying to compare cudasirecon output with that from softworx on some data we have already reconstructed with softworx. The cudasirecon is failing in a strange manner: we get a successful reconstruction but it appears to find stripes with exactly twice the expected line spacing. The the reconstructions have a small enhanced resolution but not the expected successful reconstruction.

We believe that the issue is that the initial k0 estimate is incorrectly calculated and the initial guess for the k0 position in pixels in Fourier space is incorrect. The angle is fine but the magnitude is half as big as expected.

I'm tagging @iandobbie as I'm working with him on this.

cudasirecon log (525nm)
softworx log

[tlambert03]

can you share the parameters you used as well? It's been a long time since I used softworx, and I can't remember off the top of my head, but it's conceivable that the spacings are expressed differently. I can tell you that on my OMX, for 488nm excitation, i use

k0angles=-0.804300,-1.8555,0.238800
ls=0.2035

(and in the log output for that wavelength, it settles on:

Optimum k0 angle=-0.803223, length=98.784332, spacing=0.414641 microns

so if you're starting with linespacing 0.39600 (which I pulled from your softworks log file), you might need to halve that.

maybe @linshaova, remembers whether there was a difference in the way GE expressed linespacing?

[thomasmfish]

Quick reply, thank you!

I've attached the config file (without the modified ls):
config525.txt

Why is the spacing 0.414641 if your input is 0.2035? We tried halving the line spacing and this caused it to fail (bad k0 fit) - I've just run it with a halved ls again, with the same file, to illustrate via the log below.

Halved line spacing log file (ls=0.197):
data525_recon_525.log

[tlambert03]

Why is the spacing 0.414641 if your input is 0.2035?

it is indeed confusing! 😂 I need to look back into the source code a bit to figure that seeming contradiction out.

The source code is here if you'd like to look yourself:

cudasirecon/src/cudaSirecon/gpuFunctionsImpl.cu

Lines 874 to 995 in 97af94d

	__host__ void fitk0andmodamps(std::vector<GPUBuffer>* bands,
	GPUBuffer* overlap0, GPUBuffer* overlap1, int nx, int ny, int nz,
	int norders, vector *k0, float dxy, float dz, std::vector<GPUBuffer>* otf,
	short wave, cuFloatComplex amps[], ReconParams * pParams)
	{
	int fitorder1 = 0;
	int fitorder2 = 0;
	if (nz > 1) {
	fitorder2 = 2;
	if (pParams->bBessel)
	fitorder2 = 1;
	}
	else {
	fitorder2 = 1;
	}
	float k0mag = sqrt(k0->x * k0->x + k0->y * k0->y); // in 1/um
	float k0angle = atan2(k0->y, k0->x);
	/* recalculate the overlap arrays at least this first time */
	int redoarrays = (pParams->recalcarrays >= 1);
	float x2 = k0angle;
	cuFloatComplex modamp;
	float amp2 = getmodamp(k0angle, k0mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	/* recalculate the overlap arrays every time only if recalcarrays >= 3*/
	redoarrays = (pParams->recalcarrays >= 3);
	float deltaangle = 0.001;
	float deltamag = 0.1 / (std::max(nx, ny) * dxy); // in 1/um
	float angle = k0angle + deltaangle;
	float x3 = angle;
	float amp3 = getmodamp(angle, k0mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	float amp1;
	float x1 = 0.0;
	float a;
	if (amp3 > amp2) {
	while(amp3 > amp2) {
	amp1 = amp2;
	x1 = x2;
	amp2 = amp3;
	x2 = x3;
	angle += deltaangle;
	x3 = angle;
	amp3 = getmodamp(angle, k0mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	}
	} else {
	angle = k0angle;
	a = amp3;
	amp3 = amp2;
	amp2 = a;
	a = x3;
	x3 = x2;
	x2 = a;
	while (amp3 > amp2) {
	amp1 = amp2;
	x1 = x2;
	amp2 = amp3;
	x2 = x3;
	angle -= deltaangle;
	x3 = angle;
	amp3 = getmodamp(angle, k0mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	}
	} /* the maximum of modamp(x) is now between x1 and x3 */
	angle = fitxyparabola(x1, amp1, x2, amp2, x3, amp3); /* this should be a good angle. */
	/***** now search for optimum magnitude, at this angle *****/
	x2 = k0mag;
	amp2 = getmodamp(angle, k0mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	float mag = k0mag + deltamag;
	x3 = mag;
	amp3 = getmodamp(angle, mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	if (amp3 > amp2) {
	while (amp3 > amp2) {
	amp1 = amp2;
	x1 = x2;
	amp2 = amp3;
	x2 = x3;
	mag += deltamag;
	x3 = mag;
	amp3 = getmodamp(angle, mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	}
	} else {
	mag = k0mag;
	a = amp3;
	amp3 = amp2;
	amp2 = a;
	a = x3;
	x3 = x2;
	x2 = a;
	while (amp3 > amp2) {
	amp1 = amp2;
	x1 = x2;
	amp2 = amp3;
	x2 = x3;
	mag -= deltamag;
	x3 = mag;
	amp3 = getmodamp(angle, mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 0);
	}
	} /* the maximum of modamp(x) is now between x1 and x3 */
	mag = fitxyparabola(x1, amp1, x2, amp2, x3, amp3); /* this should be a good magnitude. */
	/* if we were perfectionist we'd iterate for angle again now */
	printf("Optimum modulation amplitude:\n");
	redoarrays = (pParams->recalcarrays>=2); /* recalculate the d_overlap arrays for optimum modamp fit */
	amp3 = getmodamp(angle, mag, bands, overlap0, overlap1, nx, ny, nz,
	fitorder1, fitorder2, dxy, dz, otf, wave, &modamp, redoarrays, pParams, 1);
	/* one last time, to find the modamp at the optimum k0*/
	printf("Optimum k0 angle=%f, length=%f, spacing=%f um\n", angle, mag, 1.0 / mag);

I'll try to find out something more concrete for you, and maybe @linshaova will also have thoughts

[linshaova]

Hi everyone,

Sorry for the confusion. Line spacing output from the code should be roughly twice the number you input with --ls flag in 3D SIM. The output line space corresponds to the lowest order lateral SIM pattern's line spacing, which in 3D SIM is double the finer pattern. This was due to our effort to unify the code so that line spacing output (not input) has the same definition for all modes of SIM: 3D SIM, 2D SIM, and nonlinear SIM.

lin

[tlambert03]

thanks @linshaova.

@thomasmfish, I just double checked my softworx settings, and they are indeed very close to my cudasirecon settings (for dv files)... on softworx as well, I have a linespacing of ~208nm for the 488x/525m channel. So, I'm a bit surprised that your line spacing would be upwards of 400nm even on softworx. Are you doing anything unusual that would cause your lines to be spaced so far apart (like, are you using a low NA objective or doing something different?).

one thing I'm remember now... if I recall correctly, dv files have an inverted Y axis compared to tiff files, and if you look at the two config files in our test data here, the angles are indeed different:

cudasirecon/test_data/config

Lines 1 to 5 in 97af94d

	nimm=1.515
	background=0
	wiener=0.001
	k0angles=-0.804300,-1.8555,0.238800
	ls=0.2035

cudasirecon/test_data/config-tiff

Lines 1 to 6 in 97af94d

	nimm=1.515
	fastSI=1
	background=0
	wiener=0.001
	k0angles=0.804300,1.8555,-0.238800
	ls=0.2035

so, perhaps, drop your linespacing back down to the one closer to 200nm, and flip the sign on all of your angles

[thomasmfish]

@linshaova Thanks for your input. I believe we were expecting the 2nd order line spacing to be found and it is only finding the 1st - @iandobbie has a much deeper understanding than I do, so hopefully he can confirm whether I'm using the correct terms here. We were expecting double the magnitude for the k0s and were confused by why the line spacing wasn't consistent, so we thought there might be some scaling issue somewhere (perhaps just for tiffs - hence the name of the issue).

@tlambert03 We are using an NA of 0.9, and I've made sure the line spacings correctly match softworx for each wavelength, so I believe they should be correct.
The k0 signs are a good point - I hadn't considered that. It seemed to be getting good matches for the k0 angles but I don't believe I have tried inverting them. I am surprised it managed to fit so closely for such incorrect k0s if that is the case though. I'll have to try tomorrow and get back to you.

[tlambert03]

I am surprised it managed to fit so closely for such incorrect k0s if that is the case though.

yeah, i agree, that would be the surprising thing, but I've definitely seen it "settle" into incorrect local minima before when it was way off.

I believe we were expecting the 2nd order line spacing to be found and it is only finding the 1st

in any case, the number that the software is expecting you to enter is definitely that outer order, so, if you are using a 0.9 NA lens then your spacing in the upper 300nm range does sound about right. My guess/hope is that if you stick with that spacing, and invert the angles, it should be fine :) let us know how it goes

[thomasmfish]

Inverting the angles causes the reconstructions to fail, and I realised that we did check the magnitude against an FFT and found it matched the 1st order magnitude. At this point, I think the issue may be that it is failing to find the 2nd order, which is why it is blurrier than expected. Or it's a scaling factor. I'll spend some time looking through in more detail today.

[tlambert03]

Could you perhaps share the data?

One more question: since you're able to reconstruct this in softworx, you must be using a dv file there right? (I don't recall them supporting tiff?). Just want to know whether this is a parameter issue, or a file type issue

[iandobbie]

I think something in the code must be flipping the angles as the fits succeed and produce angles very similar to softworx and output spacings close to exactly twice the softworx output. The reconstructions we are getting end pretty sharply at about 300 nm rather than the 200 nm which made me think we were getting about 2^0.5 improvement from the coarse stripes rather than the 2x expected from a full 3d SIM reconstruction.

I guess out issue must be something else, maybe the OTFs? there is a bit of variability but the fitting amplitudes look similar to the softworx results so I am surprised about the pretty dramatic reduction of the reconstruction resolution.

Another avenue I have investigated but not got to the bottom of is possible differing scale of the Wienner parameter so we are effectively filter much more aggressively with the cuda code compared to the softworx reconstruction.

Thanks for your help and pointers so far. We are getting towards a working pipeline with much faster reconstructions than previously.

[iandobbie]

Could you perhaps share the data?

Tom will have to answer the data sharing issue, not mine to share I'm afraid

One more question: since you're able to reconstruct this in softworx, you must be using a dv file there right? (I don't recall them supporting tiff?). Just want to know whether this is a parameter issue, or a file type issue

The original data are .dv files. Tom has code that is taking the .dv files and dumping them into the recon process based on the python code in the repository but with adaptions to suit their workflow. The issue may well be file type related and I tried to build the cuda code to work with .dv files but struggled with cuda library versions etc... I am traveling for a couple of weeks and only have a mac laptop with no nvidia card so can work on this until I return.

[tlambert03]

I think something in the code must be flipping the angles as the fits succeed and produce angles very similar to softworx and output spacings close to exactly twice the softworx output.

The reason I don't think this is the case is "generally" is that I'm using very similar parameters (with dv files) in both softWoRx and cudasirecon. (Granted it's been a long time since I used softWoRx, but when I checked yesterday, the params I was using were the same as what I used to use). And when I did my original pipeline switch as you are doing, I eventually found extremely similar reconstruction results once I got the parameters right.

However, I still almost exclusively use dv files and not tiff.

That's why I'm currently less convinced by the theory that the code is wrong, and more curious about what might be different about your use case or file (eg tiff vs dv). I've never tried an air objective with low NA though, so it's possible that there's an error along those lines that I hadn't run into

Might need to see the data to be of much more help

[thomasmfish]

I have been given permission to share the data, though I'm not sure about posting it publicly. How should I get it to you?

As Ian said, we are using DV files in softworx. For cudasirecon, I am splitting the DV files by channel, saving to TIFFs and config files (the config containing settings from the DV metadata), then passing to pycudasirecon via numpy array.

[tlambert03]

You can email a link? [email protected]

I'll mention the following codebase as well:
https://github.com/tlambert03/otfsearch

The code there is absolutely hideous 🤢 😅 and 10 years old (using python 2). I wrote it before I created the pycudasirecon wrapper and never got around to updating... but that's what all my users use on our OMX to split channels/try a bunch of OTFs, pick the best one, reconstruct and merge them back together, etc... it stays with dv/mrc files all the way through (but conceptually, it should be fine with tiff as well, we just didn't have tiff support when I wrote that). Might find something to compare to in there amidst all the flotsam :)

[tlambert03]

Also inasmuch as you're ultimately using pycudasirecon, you might consider reading and splitting your dv files directly into numpy buffers (without writing to tiff intermediate) using mrc: https://github.com/tlambert03/mrc

[thomasmfish]

I'll collect it together in some kind of understandable way and send a link over, thanks!

I'll have a look at otfsearch, that sounds useful (I won't judge, don't worry!)

I am using mrc already but I decided to write to TIFF for two reasons:

1. Avoids keeping more data in memory (I found pycudasirecon didn't like getting memmaps)
2. To have intermediate files to inspect and run manually on the command line if necessary. I tested a while back and was getting the exact same result using cudasirecon from the command line with those TIFFs as passing in the arrays.

[tlambert03]

sounds good!

[tlambert03]

ok, I had a moment to look at the data (thanks for sending), and it is indeed related to the inversion of the Y axis in TIFF vs dv files that I was mentioning in #21 (comment). I opened your data525.tif file, flipped it vertically, then resaved and reconstructed with your config525.txt file (unmodified), and it worked. It's admittedly annoying figuring out how to flip the angles correctly (it's not just a sign conversion), and I also think that this code requires the angles to be expressed in exactly the correct way (for example, it's possible that 180-degree off values will not work), so you'll either need to experiment a bit with the angles, or simply flip your Y axis in numpy before saving to tiff.

Let me know if you're able to successfully reconstruct with a flipped tiff, and (if you try) whether you're able to determine the proper angles to use for a non-flipped tiff

[linshaova]

Sorry for chiming in late! Why isn't flipping the sign (not turning around by π) of k0 angle enough if vertical flip is the only difference between TIFF and DV files? I think sign-flip should work for all angles if specific angle numbers are provided (separated by comma). If only the first k0 angle is specified, then at least the first angle after sign-flipping should work.

[tlambert03]

Why isn't flipping the sign (not turning around by π) of k0 angle enough if vertical flip is the only difference between TIFF and DV files

it's a good question 😂 it's always possible I/we did something wrong here, but that was my observation so far, and haven't had time to dig deeper...

fwiw, his angles are 0.264,-1.83, 2.353 ... and from the reconstruction it looks like angle 1 and 2 were found fine, but the last one wasn't...

[linshaova]

These angles are 2π/3 apart; is this by design? I'm just curious because in all incarnations of SIM I've encountered, the pattern angles are π/3 apart; but there's nothing wrong in theory using 2π/3 angle step.

That aside, regarding the 3rd angle, would adding π to it not make any difference?

[thomasmfish]

Let me know if you're able to successfully reconstruct with a flipped tiff, and (if you try) whether you're able to determine the proper angles to use for a non-flipped tiff

I'm really sorry, but it looks like I made a mistake when sending over those files... The *_recon.tif files aren't correct (they are from using the inverted angles) but stitched_channel_output.dv was from using the correct angles k0angles=0.264228,-1.829976,2.353254. Flipping the TIFFs gives me the same output as before, with reduced sharpness compared to softworx. If you compare stitched_channel_output.dv with 20240514145542_B24_09_G3_C4B_GR_FL_SIR.dv from softworx, you'll see the difference. The output from cudasirecon is also differently scaled in intensity, but that isn't the difference I'm concerned about.

I guess this means that inverting the symbols on the angles is equivalent to flipping the array but I sent the wrong files, which confused things!

These angles are 2π/3 apart; is this by design? I'm just curious because in all incarnations of SIM I've encountered, the pattern angles are π/3 apart; but there's nothing wrong in theory using 2π/3 angle step.

That aside, regarding the 3rd angle, would adding π to it not make any difference?

@iandobbie may be able to answer the 2π/3 question but I think it's just a historical oddity as the angles should work out as equivalent to π/3 steps but in a different order.

[tlambert03]

oh, i see... so this issue isn't about a ("catastrophic") failure in reconstruction, it's more about the subtle difference in resolution bump between softworx and cudasirecon? Just to make sure we 're all on the same page here, and so @linshaova can see exactly what's being discussed here. I plotted the radial FFT plot of your raw/WF data, the softworx reconstruction (SIR) and the stitched_channel_output you provided (cudasir):

so, there does seem to be a slight bump in contrast out at the very highest frequencies in the softworx reconstruction compared to the cudasirecon, but I don't think that looks like a band has been completely missed. we of course don't have access to source code of softworx... not sure if they ever modified stuff after receiving the code from UCSF. @linshaova might have an idea (or further questions for you) on hints for why there is a slight detail in high res contrast between the two...

can you confirm that the difference in the chart above is indeed what you're concerned with here?

[linshaova]

Thanks for the clarification. So every angle worked and flipping k0 angles worked for un-flipped TIFF?

Regarding the slight lower signal strength of CudaSIR result at higher resolution shown in the graph, can't it be amended by using slightly smaller Wiener number? Like Talley said, we don't know how Softworx handle the wiener filtering and thus there's nothing so alarming about using slightly different Wiener number.

[thomasmfish]

Thanks for the clarification. So every angle worked and flipping k0 angles worked for un-flipped TIFF?

There are some not-so-great files that don't have every angle working but they seem to match softworx in terms of success.

Regarding the slight lower signal strength of CudaSIR result at higher resolution shown in the graph, can't it be amended by using slightly smaller Wiener number? Like Talley said, we don't know how Softworx handle the wiener filtering and thus there's nothing so alarming about using slightly different Wiener number.

The FFTs are an interesting comparison...

FFT of softworx image with a Wiener number of 0.001:

cudasirecon with wiener=0.001:

cudasirecon with wiener=0.0005

cudasirecon with wiener=0.0001

cudasirecon with wiener=0.00005

As you can see, the Wiener filtering appears to largely work the same for softworx and cudasirecon, with the exception that there is a halo-like effect around the central pattern that is never seen in the cudasirecon data, even at ridiculously low Wiener values (this appears to get a sharp cut-off). Unless I completely misunderstood, Ian seemed to think that this was due to missing 2nd order data as a result of a scaling issue somewhere.

My only other thought is could this be due to the OTFs being processed slightly differently? I don't have the specific parameters used when generating them in softworx, so they may not be a 100% match.

[thomasmfish]

Oops, I didn't reply to your questions @tlambert03!

oh, i see... so this issue isn't about a ("catastrophic") failure in reconstruction, it's more about the subtle difference in resolution bump between softworx and cudasirecon?

Yes, you are correct - the chart you plotted shows what we were talking about. Hopefully the FFTs for different Wiener numbers illustrate why we were concerned but perhaps there are some modifications in softworx, as you said. It would be interesting to see if they are doing something differently.

I am pretty out of my depth on this but I was wondering if it could be using a different regularization parameter? It seems like there is a much sharper falloff with the cudasirecon Wiener filtering, which is especially visible at low Wiener numbers.

[linshaova]

I'm not sure why you suspect 2nd order is missing. In the duplicated FFT below, I think the conventional WF resolution limit should be roughly within the blue circle and therefore the FFT shows doubled resolution of SIM. Could you verify with your raw data's FFT (and keep in mind that the dimension of the SIM's FFT doubled that of the raw data's FFT) that my guess was correct? Or if you tell me the xy pixel size of your raw data, I can tell you for sure if that blue circle is correct or not.

And was the area marked with red dashes what you referred to as "halo"? If that's the case, then I'd say cudasirecon results are free of that and should be considered better, especially with wiener=0.0005. wienner=0.0001 is starting to make that "halo" show up.

[thomasmfish]

I think it came from the combination of reduced high frequency resolution and the difference in k0 magnitudes, as missing the second order would align with such a difference. @iandobbie, is there anything I've missed?

Yes to the red dashed lines. I'll have to check the raw data tomorrow though, sorry!

The raw data pixel size is 0.125 with a zoomfactor of 2, so 0.0625.

[linshaova]

Given 0.125 μm pixel size in the raw data, and 0.9 NA objective (~0.3 μm resolution limit), the extent of the FFT should fill about 83% (0.125*2/0.3) of the Fourier space boundary, which is not the case shown in your FFT and I agree with you that the 2nd order does seem missing in all softworx and cudasirecon results.

However, the cudasirecon processing log you provided in the first post seems like both orders are optimally extracted (the amp numbers are all around 1.0, the ideal scenario), so I can't understand why order 2 would be missing. Curious!

[iandobbie]

...
and I also think that this code requires the angles to be expressed in exactly the correct way (for example, it's possible that 180-degree off values will not work), so you'll either need to experiment a bit with the angles, or simply flip your Y axis in numpy before saving to tiff.

I have previously found that the softworx code was very fussy about the angles and the expected 180 deg shift didn't produce a reliable fit. I think you are correct that there is an expected quadrant and the shifts in Fourier space don't work for some "equivalent" angles. I never quite got to the bottom of this but generally just calculated the 180 deg rotated angle and see if that fit with what looked like good data and then checked which angle worked and saved that into a general configs.

[iandobbie]

Sorry for chiming in late! Why isn't flipping the sign (not turning around by π) of k0 angle enough if vertical flip is the only difference between TIFF and DV files? I think sign-flip should work for all angles if specific angle numbers are provided (separated by comma). If only the first k0 angle is specified, then at least the first angle after sign-flipping should work.

I believe that the zero angle is vertical, so a sign flip would fix a reflection in X and not Y.

[iandobbie]

However, the cudasirecon processing log you provided in the first post seems like both orders are optimally extracted (the amp numbers are all around 1.0, the ideal scenario), so I can't understand why order 2 would be missing. Curious!

Tom and I spent some time with this process (Tom mostly!) and once we got it working it is clear there is less high frequency information and the log value of the stripes at double the expected value from the softwrox output made me think it was failing with regards to the fit and somehow using the low frequency stripes as effectively producing 2^0.5 resolution increase. I think this is not the issue but there is a strange issue that the higher resolution data is much more heavily attenuated with the cuda code over softworx. might be scaling and weinner filtering or some other issue.

I am traveling for another week, so wont have access to a machine allowing me to do any testing until then. Thanks for all the thoughts and I think we should have a serious look at all the components, OTF, etc and also try to get the cuda code working with the mrc files to truely eliminate any mrc to tiff issues.

[thomasmfish]

However, the cudasirecon processing log you provided in the first post seems like both orders are optimally extracted (the amp numbers are all around 1.0, the ideal scenario), so I can't understand why order 2 would be missing. Curious!

The amplitude can be greater than 1 with cudasirecon, which doesn't seem to be the case for softworx. Is there any scaling applied?

I believe that the zero angle is vertical, so a sign flip would fix a reflection in X and not Y.

I was seeing the same result (but flipped) if I flipped the TIFF as if I flipped the signs on the k0 angles...

I am traveling for another week, so wont have access to a machine allowing me to do any testing until then. Thanks for all the thoughts and I think we should have a serious look at all the components, OTF, etc and also try to get the cuda code working with the mrc files to truely eliminate any mrc to tiff issues.

I'll have a deeper look at the OTFs. I'll try and convert the softworx OTFs to see if I can use them directly. Also, I'll see if I can dig up some other test files from when the SIM was better calibrated.

[tlambert03]

thanks all for your input and patience. it would be great to have you using cudasirecon @iandobbie and @thomasmfish, so let's definitely keep digging. I'll try to dig in on my side as well and re-examine the softworx/cudasirecon differences.

and also try to get the cuda code working with the mrc files to truely eliminate any mrc to tiff issues.

one of the best things we could do here is try to drop the dependency on the IVE libraries. they are ancient, unmaintained, restrictively licensed, and we're just barely limping along by linking against the pre-compiled libraries (and I believe they may only be working on linux). I'm going to try to port my python code back to cpp so that we can finally drop that dependency, and just directly ship cudasirecon on conda-forge with mrc support.

[thomasmfish]

Thank you @tlambert03 and @linshaova! You've been incredibly helpful.

The code I'm using expands upon pycudasirecon, taking DV files, splitting them (saves as TIFFs, though not strictly necessary) and rejoins them, with some config handling. I'm planning on making it open source ASAP but this depends on my manager, who is on holiday. I'm not sure if it will be helpful at all, but I can let you know when it is available.

I'm personally not too fussed about reading DV files directly, but rather about whether any parameters are (or processing is) different with TIFFs, but I either way would fix the problem for me.

[iandobbie]

one of the best things we could do here is try to drop the dependency on the IVE libraries. they are ancient, unmaintained, restrictively licensed, and we're just barely limping along by linking against the pre-compiled libraries (and I believe they may only be working on linux). I'm going to try to port my python code back to cpp so that we can finally drop that dependency, and just directly ship cudasirecon on conda-forge with mrc support.

Yes getting rid of the IVE dependencies would be great, MRC files are not that complicated and supporting the OMX (API?) defined extended header fields for SIM would be good as this allows specification of both and excitation and emission wavelength etc...

I am still awaiting news on a grant that would have a full time programmer who could definitely contribute to such an effort. I'll let you know if we get anywhere.

[linshaova]

Tom and I spent some time with this process (Tom mostly!) and once we got it working it is clear there is less high frequency information and the log value of the stripes at double the expected value from the softwrox output made me think it was failing with regards to the fit and somehow using the low frequency stripes as effectively producing 2^0.5 resolution increase. I think this is not the issue but there is a strange issue that the higher resolution data is much more heavily attenuated with the cuda code over softworx. might be scaling and weinner filtering or some other issue.

I'm not seeing the big difference between the Softworx and cudasirecon regarding high-resolution attenuation, especially after lowering Wiener in the latter (do you, @tlambert03?). I see attenuation in both results and the only possibility that I can think of is the following. There is a Fourier space boundary, calculated to be equal to the theoretical conventional resolution, used in the code's final assembly steps. If that boundary were calculated wrongly somehow then one could see the attenuation such as in this example. However, the only way this simple number can be miscalculated is if the input wavelength and NA parameters (plus XY pixel size) were wrong. Can it be confirmed that the correct wavelength, NA and pixel size numbers are fed into the program (especially for TIFF because there's no header that contains the wavelength)?

[linshaova]

p.s.: The "log value of the stripes at double the expected value from the softworx" turns out to be non-issue: it's just a different convention of expressing the same value, which I hope I did explain clearly earlier on.

[thomasmfish]

Hi @linshaova, the config I shared has all the settings used (linked again here: config525.txt). Obviously the Weiner number changed for the above FFTs, but the rest should be consistent. When compared to the values in the softworx log, they seem correct. It could be useful to have an option to print out all the parameters for checking this.

[linshaova]

Thank you for sharing this again.

Could you uncomment this line in gpuFunctionImpl.cu temporarily and rebuild and rerun?

cudasirecon/src/cudaSirecon/gpuFunctionsImpl.cu

Line 611 in 97af94d

// printf("In makeoverlaps(), order1=%d, order2=%d, k0x=%f, k0y=%f, rdistcutoff=%f, zdistcutoff=%f pixels\n", order1, order2, kx/dkx, ky/dky, rdistcutoff/dky, zdistcutoff);

I'd like to see what the rdistcutoff number comes out to be; it should be around 200 for 512x512 raw data dimension.

Also, is the background really 200? If you use a sCMOS camera, the background should be around 100 for almost all models.

[thomasmfish]

Using the OTF from softworx saved as a TIFF then processed with cudasirecon

Wiener 0.0005

Definitely different filtering (this is visible in the reconstruction too).

In terms of the background, I've no idea. I'm just using the value from softworx. I've tried with background set to 0 and didn't see a difference.

I haven't tried building it myself, I'm using the conda version.

[linshaova]

Never mind; actually, I just realized your results make complete sense based on the SIM line spacing (0.394 um) you chose and nothing is wrong with either result, softworx or cudasirecon. Let me explain.

Conventional resolution limit at 525 nm 0.9 NA should be around 525/2/0.9=291 nm (I'm using Abbey limit number). Nyquist-sampling pixel size would be 145 nm for this resolution limit.
Pixel size in use in 125 nm, and therefore the extent of the resolution limit should be about 125/145=86% of the maximum Fourier space frequency present in your raw data based on the sampling rate of 125nm per pixel. (Note: if using Rayleigh resolution limit, this percentage is even smaller; ~70%)

If you used 291 nm SIM line spacing and double the resolution, then this 86% number would hold in the final SIM result because of both the doubled resolution limit of the data and the halved pixel size.

However, because 394 nm line spacing is used and therefore only 74% (=291/394) resolution enhancement is achieved (as opposed to 100% or doubled resolution), the FFT extent of your SIM result should be about 63% (74% * 86%; or smaller if Rayleigh limit were used) of the maximum frequency in the final result, which is not far from what show in the FFT images, especially with the lower Wiener number in cudasirecon results. I try to make this point easier to understand by using this sketch, in which the green line is about 63% length of the red line and therefore consistent with the resolution gain derived from the 394 um line spacing:

[linshaova]

Update: I obtained the real data via Talley. From measuring the radially averaged OTF, I can tell that the conventional resolution limit is about 77% of maximum frequency defined by the 125 nm pixel size. In the raw data, the maximum frequency thus defined is 1/(512*.125)*256 = 4 cycles/um, and therefore the .conventional resolution is about 3.06 cycles/um, roughly in the middle between Abbey and Rayleigh limit numbers (which makes sense).

Using this number and the line spacing number of 0.394 um (translated into 1/.394 = 2.538 cycles/um), the outer edge of the SIM resolution limit would thus be 3.06+2.538 = 5.598 cycles/um, and in terms of percentage of this limit over the maximum frequency of 8 cycles/um in the SIM reconstruction (because of halved pixel size, 4 becomes 8), it's roughly 70% and consistent with the results' FFT images shown above.

[linshaova]

One side note... I looked at the OTFs generated by softworx, the boundary (corresponding to the conventional resolution limit) is carved out incorrectly. It should be along the red dash line, as opposed to the green dash line that the results indicate:

This is not super critical, but if you want to do this correctly, I suspect you specified too small a pixel size or something when the OTF was generated. You should use 0.125 um as the pixel size if that's not what you used already.

[thomasmfish]

Thanks for investigating - it sounds like the reconstructions are correct then, though the filtering is different for high-frequency details. I have been on a course the past couple of days so haven't been able to look into this in any detail, but I have managed to grab some older data (hopefully from when the microscope was better aligned), which I will compare to see whether the difference persists in a consistent manner.

One side note... I looked at the OTFs generated by softworx, the boundary (corresponding to the conventional resolution limit) is carved out incorrectly. It should be along the red dash line, as opposed to the green dash line that the results indicate:

This is not super critical, but if you want to do this correctly, I suspect you specified too small a pixel size or something when the OTF was generated. You should use 0.125 um as the pixel size if that's not what you used already.

I didn't do the initial OTF generation via softworx, so I can't be sure. However, I can't see that pixel size is set as part of OTF generation (at least in cudasirecon), though it could get the pixel size from the DV metadata. Hopefully the ones I generated via cudasirecon are better(?).

I've not been able to get a good understanding of the leavekz values (particularly the third value). I've been reading around but haven't found anything that goes into those values specifically. Would you be able to point me in the direction of something that explains how to set leavekz?

[tlambert03]

here is some extremely clean (simulated) data, with reconstructions in cudasirecon at weiner 0.001 and 0.0001 as well as softworx 0.001 with "triangle" vs "standard" apodization medthods

https://www.dropbox.com/scl/fi/odglvihvgbuuu5tejw6yn/sim_test_data.zip?rlkey=2yradd7ra1cvky7arayj2kgzh&dl=0

it's useful for the comparison since the data is nearly "perfect" ... but then again it's synthetic, so has caveats.

the differences are definitely subtle, and indeed can probably be attributed to slight differences in apodization/weiner processing rather than fundamental differences in band detection, etc...

from left to right: cudasirecon0.0001, cudasirecon 0.001, softworx 0.001 triangle, softworx 0.001 standard

[tlambert03]

I've not been able to get a good understanding of the leavekz values (particularly the third value). I've been reading around but haven't found anything that goes into those values specifically. Would you be able to point me in the direction of something that explains how to set leavekz?

from the readme:

-leavekztakes three integers as arguments. Purpose of this flag is to zero out the region outside of the OTF support. The first two numbers correspond to the two pixels on the positive k_z axis of the order-1 OTF, between which the order-1 OTF support intersects the k_z axis. The third number corresponds to a pixel on the positive k_z axis, between which and the origin the order-2 OTF support intersects the k_z axis. With these other related numbers such as NA and wavelengths, the program could calculate what's inside and what's outside the OTF supports and then zero out the outside parts.

but @linshaova could probably answer any additional things if that explanation leaves open questions

[iandobbie]

I have been thinking about where the differences between the two processing pipelines might come from. I understand Lin's arguments about the expected resolution but am trying to track down the significant differences we see.

Obviously the line spacing is a red herring. With this eliminated I suspect that the OTF processing might be the most significant difference.

1. NA, Tom is setting the correct (0.9 NA) for both OTF creation and for the reconstruction. This parameter is not directrly exposed by the softworx interface and I wouldn't be surprised if it is set to 1.4 by default, not read from the objective database, for both the OTF creation and for the reconstruction.

2. leavekz processing. I am not sure exactly what is being set by Tom in the current processing but these were historically hand tuned in the OTF creation process, requiring different values for different wavelengths. I am not sure if Tom has none set or is using some set value, but I am pretty sure it could do with some tuning.

3. Background. This should be properly measured and then set to that value. I have no idea what the realistic background is on the cameras, but it could easily be 200 as the system is running on Andor iXon EMCCDs.

4. Bead size compensation. My experience is that not using the bead size compensation you can often end up with larger than 1 amplitudes in the fitting process. I would not be surprised if the softwrox OTFs have compensation turned on and the more recent cuda ones don't.

I think my next step is to try and setup both cuda and softworx reconstruction workflows and follow them through closely to see how the various steps compare. I am still away till the end of the week.

[thomasmfish]

from the readme:

Of course I missed the place that was right in front of me! I think I need to spend more time reading around it to understand that fully but I'm glad I have something to go by now. Thanks for highlighting that section for me.

the differences are definitely subtle, and indeed can probably be attributed to slight differences in apodization/weiner processing rather than fundamental differences in band detection, etc...

There's definitely a sharper drop-off around just before 10 in the radial FFT plots of cudasirecon results, which I guess is what we're seeing here too.

2. leavekz processing. I am not sure exactly what is being set by Tom in the current processing but these were historically hand tuned in the OTF creation process, requiring different values for different wavelengths. I am not sure if Tom has none set or is using some set value, but I am pretty sure it could do with some tuning.

I have set values per-wavelength but as I've never done it before, so I wouldn't be surprised if they could be improved. The reason I asked about the third leavekz value is that I've just left it as 5, matching how it is in softworx. In case it's useful:
452: leavekz=5,7,5
525: leavekz=5,6,5
605: leavekz=3,5,5
655: leavekz=3,4,5

4. Bead size compensation. My experience is that not using the bead size compensation you can often end up with larger than 1 amplitudes in the fitting process. I would not be surprised if the softwrox OTFs have compensation turned on and the more recent cuda ones don't.

I have been using the setting nocompen=False, but it turns out something is a little broken somewhere and it's getting set to true, so that needs fixing!

I am still away till the end of the week.

Thank you so much for taking this time, especially while you're away.

[thomasmfish]

Probably not important but just in case, I've fixed the nobeadcomp setting issue - it was a silly mistake when parsing the config. Now the amplitudes are more comparable. The amplitudes do seem to be larger with softworx.

Softworx:

----- Processing Time 1, Channel 605, Direction 1
  k0guess[direction 1]= (135.835907, -41.062965)
  k0pixels= 141.906876,  k0mag= 2.217295,  alpha= 1.117470, beta= 0.400812, betamin= 0.304973
  after search: k0[direction 1] = (135.932495,-36.952301)
  *** WARNING:  Best fit for k0 is 4.111799 pixels from expected value.
  k0pixels= 140.865585,  k0mag= 2.201025,  alpha= 1.117470, beta= 0.397705, betamin= 0.301972
  Modulation Amplitude angle= -0.264112, length= 141.223282, amplitude= 0.737633, phase= 2.636387
  Optimum k0 angle= -0.264112, length= 141.223282, spacing= 0.453183 microns
  k0pixels= 141.223282,  k0mag= 2.206614,  alpha= 1.117470, beta= 0.398772, betamin= 0.303003
  after refine: k0[direction 1] = (136.326309,-36.866695)

cudasirecon:

k0guess[direction 0] = (68.490959, -18.530493) pixels
Initial guess by findk0() of k0[direction 0] = (67.957642,-18.471130) pixels
before fitk0andmodamp
 In getmodamp: angle=-0.265392, mag=1.100362, amp=0.573444, phase=3.132662
 In getmodamp: angle=-0.264392, mag=1.100362, amp=0.594862, phase=3.088306
 In getmodamp: angle=-0.263392, mag=1.100362, amp=0.594078, phase=3.053789
 In getmodamp: angle=-0.263928, mag=1.100362, amp=0.597336, phase=3.071077
 In getmodamp: angle=-0.263928, mag=1.101925, amp=0.657483, phase=2.775250
 In getmodamp: angle=-0.263928, mag=1.103487, amp=0.675747, phase=2.540723
 In getmodamp: angle=-0.263928, mag=1.105050, amp=0.634787, phase=2.360375
Optimum modulation amplitude:
 In getmodamp: angle=-0.263928, mag=1.103194, amp=0.676504, phase=2.580526
 Reverse modamp is: amp=2.282641, phase=2.580526
 Combined modamp is: amp=1.480089, phase=2.580526
 Correlation coefficient is: 0.544398
Optimum k0 angle=-0.263928, length=1.103194, spacing=0.906459 um
 In getmodamp: angle=-0.263928, mag=1.103194, amp=0.513559, phase=1.410684
 Reverse modamp is: amp=1.508000, phase=1.410684
 Combined modamp is: amp=0.804229, phase=1.410684
 Correlation coefficient is: 0.583572
best fit for k0 is 0.246% from expected value.

Though those are using different OTFs (same PSFs). Using the same OTF softworx used, it gets a much lower amplitude:

k0guess[direction 0] = (68.490959, -18.530493) pixels
Initial guess by findk0() of k0[direction 0] = (67.954758,-18.472366) pixels
before fitk0andmodamp
 In getmodamp: angle=-0.265420, mag=1.100324, amp=0.286672, phase=-3.060912
 In getmodamp: angle=-0.264420, mag=1.100324, amp=0.295982, phase=-3.099719
 In getmodamp: angle=-0.263420, mag=1.100324, amp=0.294569, phase=-3.129492
 In getmodamp: angle=-0.264053, mag=1.100324, amp=0.296738, phase=-3.111754
 In getmodamp: angle=-0.264053, mag=1.101886, amp=0.328642, phase=2.863258
 In getmodamp: angle=-0.264053, mag=1.103449, amp=0.340743, phase=2.622633
 In getmodamp: angle=-0.264053, mag=1.105011, amp=0.322861, phase=2.440904
Optimum modulation amplitude:
 In getmodamp: angle=-0.264053, mag=1.103301, amp=0.340779, phase=2.642712
 Reverse modamp is: amp=1.274480, phase=2.642712
 Combined modamp is: amp=0.469582, phase=2.642712
 Correlation coefficient is: 0.517095
Optimum k0 angle=-0.264053, length=1.103301, spacing=0.906371 um
 In getmodamp: angle=-0.264053, mag=1.103301, amp=0.280904, phase=1.377894
 Reverse modamp is: amp=0.761161, phase=1.377894
 Combined modamp is: amp=0.320580, phase=1.377894
 Correlation coefficient is: 0.607492
best fit for k0 is 0.241% from expected value.

Inspecting the OTFs, the softworx one is definitely less well trimmed (though they're both not great), and has pixel values that are slightly more than double that of in the cudasirecon OTFs. I'm not sure what could be causing that but I could manually adjust the softworx values to match and see if that helps.

[thomasmfish]

I've run this again with the data I've been told is better. I don't think the OTF is great on some channels but the 525 is looking good and has given good results, so I'm going to focus on that. Here's the FFT of two images that look a much closer match and don't have the blurring we've been seeing. Much more of what I called a "halo" on both, though definitely more significant with softworx. I have no idea why:

1. softworx seems to be more resilient to lower OTF quality
2. What this "halo" is (it doesn't just seem to be noise) and why is more prevalent in the softworx output

Softworx:

Cudasirecon:

Here are the log files in case that's useful:

softworx log

cudasirecon log

config for cudasirecon

I still need to try the simulated data but that's for another day.

[thomasmfish]

Hi all, sorry for the radio silence on this issue. Ian and I have been investigating things and it looks like the main difference is that softworx has no way to specify the na or nimm parameters. However, I'm not sure that this is the entirety of the issue and it is hard to test since our system is has such different values.

One thing that I was unsure about is the apodization, as the options section of the logs we get from softworx contains:

  -inputapo 4
  -triangleapo

I'm not sure how those settings would map to cudasirecon, or if one of those settings overrides the other in softworx. However, it looks like in cudasirecon, m_myParams.apoGamma is never used (just passed to const_pParams_apoGamma) and apodizeoutput is always set to 2 (since the setting gammaApo has a default, see 1040 of cudaSirecon.cpp). Could this be affecting the output, perhaps more noticeably than for higher NAs?

I also wanted to mention the project I've been working on, called PySIMRecon, in case it is of interest to you. The idea is to wrap cudasirecon/pycudasirecon with Python to allow the setting of default values that match a microscope, along with OTFs, so that it can be easily used for automatic processing. It handles splitting and recombining DV files and has a couple of other tools in it too. Also, on a personal note, I am moving on from my current job at the end of this month, so I unfortunately won't be able to continue the project. However, I will be installing PySIMRecon as part of the automatic processing pipeline we use, so it should continue to be maintained and developed.