win: patch from OSGeo4W applied

[landam]

This PR applies https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/patch (author: @jef-n)

[neteler]

A dependency yet missing?

From the CI log:

...
configure: error: *** couldn't find libpng-config
Error: Process completed with exit code 1.

[nilason]

A dependency yet missing?

From the CI log:

...
configure: error: *** couldn't find libpng-config
Error: Process completed with exit code 1.

Not a dependency, but https://github.com/OSGeo/grass/blob/main/mswindows/osgeo4w/libpng-config (see #2679).

[jef-n]

Note that build_osgeo4w.sh and osgeo4w.yml are GRASS things, that are not used in osgeo4w - they were just aligned with package.sh, which actually is.

[landam]

After reintroducing mingw-w64-x86_64-libpng the CI pass successfully.

@jef-n Please could you elaborate why you are removing this dependency in the workflow (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/patch#L9)? On the other side it's installed in package.sh (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/package.sh#L101)

[jef-n]

After reintroducing mingw-w64-x86_64-libpng the CI pass successfully.

@jef-n Please could you elaborate why you are removing this dependency in the workflow (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/patch#L9)? On the other side it's installed in package.sh (https://github.com/jef-n/OSGeo4W/blob/master/src/grass/osgeo4w/package.sh#L101)

um, osgeo4w has libpng. not sure why it was only removed from build_osgeo4w.sh. As said I don't use that. Maybe I wanted to replace msys libpng with's OSGeo4W's png, replaced it in both and failed to re-add it in build_osgeo4w.sh, when I found that it was not working. But I don't recall… In OSGeo4W it's just a patch to an unused file ;)

[jef-n]

There was a change in libpng that broke our copy of libpng-config - adapted.

[landam]

Any idea why osgeo4w CI is failing on tests while compilation is successful? It seems to be unrelated to this PR...

[echoix]

There were a lot more failed files, like 58% of files passed, instead of 81%

[nilason]

At least
OpenMP issues need to be addressed for 8.4.1, I’ll add backport label for this PR as not to miss this.

[echoix]

The conflicts in the PR I might have time this weekend to address them, it's not too long, it's the sorted lines that need to be chosen with respect to the changes this PR made

[echoix]

I solved the conflicts by sorting the two conflicting files of the PR in same way as #4563, then comparing the two sides of the merge making sure changes present in the PR (on GitHub) were still present even if moved. It was all ok, nothing is lost.

[echoix]

osgeo4w installs libomp.dll:
https://github.com/jef-n/OSGeo4W/blob/f8ada2fe6637b1abde0a361fa8b5b2089a82e33d/src/grass/osgeo4w/package.sh#L104

which is probably why configure hooks up on it.

Perhaps

-/mingw64/bin/libgomp-1.dll
+/mingw64/bin/libomp.dll

in

grass/mswindows/osgeo4w/package.sh

Line 127 in 3f61cd7

/mingw64/bin/libgomp-1.dll

could be the solution?

I've reread on a fresh mind. Isn't libomp the llvm OpenMP runtime library (https://packages.msys2.org/packages/mingw-w64-x86_64-openmp, files section at the very end of https://packages.msys2.org/packages/mingw-w64-x86_64-llvm-openmp)

And libgomp the gcc OpenMP library
(https://packages.msys2.org/packages/mingw-w64-x86_64-omp, files section at the very end of https://packages.msys2.org/packages/mingw-w64-x86_64-gcc-libs)

(And libiomp the intel OpenMP one, not in msys2).

So, following the often linked advice that multiple OpenMP runtimes shouldn't be mixed (https://cpufun.substack.com/p/is-mixing-openmp-runtimes-safe), and that we are using the gcc toolchain, I think we should be removing all references to libomp.dll, and go all-in with libgomp-1.dll, and work until fixing all the issues.

[echoix]

I've implemented the suggested changes here: echoix#285

But since our CI doesn't use the package.sh script, I've reimplemented the patches that I've been waiting and wanting to do after this PR is merged here: echoix#286

It showed that we were missing some dlls for lapack and libblas. I didn't find (only on my phone) what branch I had solved the issue that the msys2 couldn't run the Linux print_versions.sh, where some extra paths need to be added so an msys2 shell finds inside the OSGeo4W install.

At least, in these CI runs, it uploads the compiled package that can be downloaded and installed with the OSGeo4W installer (pointing the download dir to a dir with the package from the artifact) for local testing. We could test if the two or three bugs with OpenMP linked are solved by that

[nilason]

OpenMP related fix (hopefully) is added with jef-n/OSGeo4W@06d4d83 and jef-n/OSGeo4W@c678c71.

[echoix]

Conflicts to solve

[nilason]

OpenBLAS issue for main has been addressed with 9984205, which doesn't need to backport.

OpenMP issues have been addressed (as mentioned above), but doesn't need changes in GRASS source (and thus no need to backport).

I removed backport label for this PR, as the build issues have likely been addressed.

[nilason]

@landam Could you please check if the win build server is working, as there has been no build log since end of August September.

[echoix]

I'm still processing the changed patch 16 hours ago in the OSGeo4W repo

[echoix]

In the last commit I'm keeping (for now) the --with-libpng=${SRC}/mswindows/osgeo4w/libpng-config \ line that was removed. I think it might be a merging/rebase conflict that lost it, as I would have expected that line to be changed to use ${SRC} instead of ${PWD} like the other places in the file that got updated for that. If it is expected, then we should also remove it from our parallel script.

[echoix]

I've gone through the remaining conversations to resolve. Like I mention in these, I would really like to have this finished, so I could resume some other work that I rebased on this (as a couple weeks before this PR I already did a similar effort and continued on this), but I've been waiting 4 months since. Recently we solved one of the blockers, so there's not much left to think here.

[echoix]

The first one, PYTHONPATH, is needed, I often had to add it to make windows tests work in CI (like when working hard to get pytest running), but the second one I don't know

[echoix]

We could add a separate issue for this, as we're mostly only applying the patches that have been used upstream. Something quite simple that we've spent 4 months on, so I would hope to not stretch it further if no easy consensus comes.

[echoix]

Is this exact script really used for the NSIS installer? This script is used only by OSGeo4W (yet) from the best of my knowledge. I have some work prepared even before this PR was filed to use this script for our CI, to be really closer to what OSGeo4W packages, and could upload the artifacts that could be tested locally, and use ccache (that is already scripted here, we only have to download and upload the cache in CI). To not duplicate work, I was patiently waiting for this PR to be merged before continuing, but I would really want to continue to allow shaving some time off of the longest CI job that we have.

[echoix]

See #4121 (comment):

In the last commit I'm keeping (for now) the --with-libpng=${SRC}/mswindows/osgeo4w/libpng-config \ line that was removed. I think it might be a merging/rebase conflict that lost it, as I would have expected that line to be changed to use ${SRC} instead of ${PWD} like the other places in the file that got updated for that. If it is expected, then we should also remove it from our parallel script.

[jef-n]

if this is merged, DLLS (line 109/10) can be removed