pre-compiled Windows version with new features

[peter-mri]

While we are able to use nlsam_denoise on linux (and I've managed to get it working on Compute Canada as well, which is a cloud cluster we can take advantage of), I'll be interested in making it working on Windows as well (some of our users use ExploreDTI on Matlab and this would allow them to run this pre-processing all on one platform).

The pre-compiled version available on GitHub seems to be from previous iteration of work, so it doesn't support the automated estimation. I have so far not been able to install (and make things work) under Python on Windows - it seems that despite no errors during installing packages, spams/spams_wrap is missing some DLLs (?)

Is there any trick you would be to recommend to get this installed, or, even better, any chance you might be able to generate updated version of pre-compiled exe file?

We have successfully integrated your NLSAM method into processing of our DTI data. Especially your recent addition of automating of estimating all parameters helps a great deal, and we are very grateful your work is available!

[samuelstjean]

Unfortunately the all in one version are likely dead for now because of this joblib/loky#236. It did made stuff easier for me since I could remove a few hacky workarounds, but it also meant I can't really fix them now until the downstream lib fixes it, which would seem not soon.

Although it s now possible to install this https://pypi.org/project/spams-bin/#files to get the libraries, I just never bothered to put all that in one once again, I'll see what I can do tomorrow about it if possible. They also depend on the mkl libs for computing (likely the missing dll I'd wager), such as available in the numpy/scipy version of conda, but unfortunately there does not seem yet to be an easy automated way to shove the libs inside the install wheel like on mac and linux.

If you want to install them separately, it's this version https://github.com/samuelstjean/spams-python/blob/master/.github/workflows/cibuildwheel.yml at line 37, although I suspect grabbing the mkl files from conda itself will likely work even if they are more recent.
Basically, this is unfortunately unusable by itself as of now on windows, but relies on the hidden mechanic that the user has anaconda and uses their mkl build of numpy, which was the only sane way to install it a few years back. Other installs may not have those files, hence the missing dll I think.

I could also re-give it a go to see if it magically got fixed now, seems like pyinstaller may have worked on something related maybe.

[peter-mri]

Thanks for the quick reply (looks like your Utrecht email is no longer working...? I got email bouncing back). I'd appreciate if you look into all-in-one version, but understand it may not be possible.

Yes, I think the missing DLLs are related to mkl, which Intel has changed quite a bit in the last few years apparently.

I've been trying to get things installed with pip, not conda (I'm noob enough with python to not understand the difference). I always find it a bit frustrating experience to work with python tbh, part of learning process... Anyways, I was able to install spams-bin with just pip, but like I said, when I attempt to run the code, it fails... perhaps I'm trying to run it the wrong way?

c:\Users\Peter\MicroBrain-main>py C:\Users\Peter\AppData\Roaming\Python\Python37\site-packages\nlsam\denoiser.py
Traceback (most recent call last):
File "C:\Users\Peter\AppData\Roaming\Python\Python37\site-packages\nlsam\denoiser.py", line 17, in
import spams
File "C:\Users\Peter\AppData\Roaming\Python\Python37\site-packages\spams_init_.py", line 2, in
from .spams import *
File "C:\Users\Peter\AppData\Roaming\Python\Python37\site-packages\spams\spams.py", line 6, in
import spams_wrap
File "C:\Users\Peter\AppData\Roaming\Python\Python37\site-packages\spams_wrap_init_.py", line 2, in
from .spams_wrap import *
File "C:\Users\Peter\AppData\Roaming\Python\Python37\site-packages\spams_wrap\spams_wrap.py", line 19, in
from ._spams_wrap import *
ImportError: DLL load failed: The specified module could not be found.

Any and all suggestions are appreciated!

[samuelstjean]

Ah yes my old emails from utrecht all got axed when I left, I am using my new ualberta one for now. The main difference is that pp just install python packages, while conda can install a whole bunch of other stuff in addition, and that's what I use for building things on windows since I can skimp on the linking libs.

In the meantime I tried something which can be downloaded here https://github.com/samuelstjean/spams-python/suites/3593308241/artifacts/86579512, it contains some dll and will try to download the rest as needed, which are hopefully the right versions. I don't have a windows computer set up close by now unfortunately, but if that does not solve it I guess I'll have to shove in manually the needed files. It should not be 'too complicated', just a bit tedious for me since it involves a bunch of trials and errors until all the correct dlls are included in the end.

The main problem is that finding the dll you directly depend on is doable, but the tools do not seem to be able to find automatically the dlls that those dlls depend on, hence it breaks and needs hand fixing. The conda version just happens to also use those libs for other stuff at the same time, hence we can rely on them being present for that, even if the versions are different (unless they change the name, which they seems to have done in 2021). Hence this version should fetch an old build with the old names normally.

[peter-mri]

Hm, can't find you with ualberta people search (yet)... Anyways, thank you, I will give this a try and keep you posted - although I have processed the current batch of data on Compute Canada, so this would be more of a "nice to have" option in the future - even if it doesn't work, probably not worth going out of your way to go through the tedious process... Obviously there aren't too many other users asking for this option.

[samuelstjean]

Normally this build of spams should work https://github.com/samuelstjean/spams-python/suites/3630670560/artifacts/87875714, at least from my very simple test, more details here samuelstjean/spams-python#16
If everything seems fine I'll try to make it official, although I am a bit worried with potential file naming clashes.

[peter-mri]

Hm, perhaps that is only for Win10? I'm still on Win7 (I'm getting behind times...)

pip install spams_bin-2.6.3-cp39-cp39-win_amd64.whl
ERROR: spams_bin-2.6.3-cp39-cp39-win_amd64.whl is not a supported wheel on this
platform.

[samuelstjean]

That is possible, or the other thing is that the cp39 means it's for python 3.9, there is also a 3.7 and a 3.8 version in the archive normally (and a mac and linux version).
Internet chooses the right one for you, but for manual install you need to specify the one according to your installed version of python.

I'm just using some online tool to build it, hopefully it does support win7 and up, but usually windows stuff is pretty good at running on older platform.

[peter-mri]

You were right, just needed to use the appropriate file for my version of Python. Unfortunately, I still get the same error about missing DLLs...

[samuelstjean]

Hmm well I'll try to have a real look over the weekend then.

[peter-mri]

At this point, it's really not a "need", just potentially nice to have. Hopefully there are more fun things to do over a weekend!

[samuelstjean]

Well normally I think I have a working version for spams here, just need other people to confirm it works for them I guess samuelstjean/spams-python#16

From there I can probably make a new(er) version for good, but it will still require a base install of python, and the rest is automatic. As for the all in one build I was using before, until someone else fixes the things I am using, I don't think it will be possible. And it doesn't seem to be super high priority either for them, likely because it is now rather simple to install all that stuff unlike 5 years ago I'd guess.

[samuelstjean]

Here, normally this https://github.com/samuelstjean/nlsam/suites/3722125282/artifacts/91062601 is the latest version. It should also fetch a prebuilt spams-bin version if I made stuff correctly, in the worst case they are linked in the previous post.

[samuelstjean]

Well partly good news, I think I have something for windows only. It's bit messier than the previous version since now everything is in a folder, but it has the advantage of working. https://we.tl/t-wBkWShBDPj Usage is also slightly different, but I added a quick readme to the files. It you think that's an acceptable solution (it currently uses the latest version with multishell and everything), I'll make it official in that case.

[samuelstjean]

Happy to say there is now an official new release with the latest things all included. It's available as either a binary install requiring python (including for the new mac arm processors) or as a standalone version.

The only caveat is that this standalone version is using a different backend for the multiprocessing (dask instead of loky, until they hopefully fix it joblib/loky#236, which is unfortunately slower than a direct install. And there is no mac arm version either (until there is a good way to publicly run and test it, since I do not have access to one of those), so it should run, but also be slower than a native install.

[samuelstjean]

Well there is now a new(er) version which is not using frozen binaries anymore but a portable install and a script to run the whole thing. It's a bit less pretty than before, but it does normally work as intended without workarounds like before.