.NET single-file deployment behaviour has changed with the libsodium v1.0.19.1+ NuGet package

[samuel-lucas6]

In .NET, you can bundle the runtime and libraries into a single executable, meaning users only need the executable to run your application (rather than needing to install the .NET runtime or have libraries in the same folder).

I've just been trying to create a new release of a project and have encountered an issue when doing a self-contained, single-file deployment for macOS (osx-x64 and osx-arm64). win-x64, win-x86, win-arm64, linux-x64, linux-arm64, and linux-arm don't seem to be affected. However, linux-musl-x64 has the same problem as macOS.

With v1.0.19 of the NuGet package, the libsodium library is bundled into the executable. However, with v1.0.19.1 onwards, it's not. You get a separate libsodium.dylib file (for macOS) or libsodium.a file (for linux-musl-x64) that needs to be kept with your executable for the application to work.

v1.0.19 was published to NuGet on the 19th September 2023, so a change after that date seems to have affected this functionality.

I unfortunately can't downgrade the libsodium package because my .NET libsodium binding enforces the latest version.

Steps to reproduce

1. Download the latest .NET 8 SDK installer.
2. Create a folder to house the project and navigate to it in the terminal.
3. Create a new console project with dotnet new console --framework net8.0.
4. Install the libsodium NuGet package with dotnet add package libsodium --version 1.0.19.1.
5. Do a self-contained, single-file publish for macOS with dotnet publish -c Release -r osx-x64 -p:PublishSingleFile=true -p:PublishTrimmed=true -p:PublishReadyToRun=true -p:IncludeNativeLibrariesForSelfExtract=true --self-contained true.
6. Navigate to the printed path (bin\Release\net8.0\osx-x64\publish\), and you'll see there's a libsodium.dylib file.
7. Delete the files in this folder and return to the original project directory.
8. Change the libsodium package version with dotnet add package libsodium --version 1.0.19.
9. Use dotnet publish again as before.
10. Navigate to the printed path again, and you'll see there's no libsodium.dylib file. This is what you want.

[samuel-lucas6]

The change for v1.0.19.1 happened between September 2023 and May 2024 (8 months ago). This is equivalent to Commits on Jan 4, 2024 => Commits on May 2, 2024.

Could it be related to a Zig update? There were some changes to the builds for macOS (80aca14), but it looks like they were reverted (08fa6d1), and I think linux-musl stayed the same besides c9a92ea.

CC'ing some people who've been involved in .NET PRs here: @nil4, @MartyIX, @ektrah, @alexrp, @js-9, @bfren, @MichalPetryka, @SeanMollet, @enclave-alistair (sorry for bothering you)

[ektrah]

It's a bit difficult to diagnose why -p:IncludeNativeLibrariesForSelfExtract=true does not include the native library in some cases without knowing the mechanics of this flag under the hood. Maybe it's better to open an issue in a .NET repo to attract the right experts?

[ektrah]

Well, it's actually not that difficult. The IncludeNativeLibrariesForSelfExtract flag is passed to the GenerateSingleFileBundle target, which uses the GenerateBundle task, which invokes the Bundler class.

A file will be excluded by the ShouldExclude method if it is a native library and the file name is libhostfxr.dylib or libhostpolicy.dylib. This is not the case for libsodium.dylib.

The file type is detected by the InferType method using the IsNativeBinary method. When publishing for macOS, the IsMachOImage method checks whether the first four bytes of the binary are one of the magic numbers defined in the MachMagic enum.

runtimes\osx-x64\native\libsodium.dylib in the 1.0.19 nupkg starts with CF FA ED FE (0xfeedfacf). In 1.0.19.1, it starts with CA FE BA BE (0xbebafeca). These are both defined (MachHeader64CurrentEndian and FatMagicOppositeEndian, respectively).

So I have no idea. It's a bit difficult to diagnose 😸

[nil4]

The macOS library files changed between 1.0.19 and 1.0.19.1. In the earlier package, the .dylibs were architecture-specific binaries:

> curl -fsSL -o 19.zip https://www.nuget.org/api/v2/package/libsodium/1.0.19 && unzip 19.zip -d 19
> file ./19/runtimes/osx-arm64/native/libsodium.dylib
./19/runtimes/osx-arm64/native/libsodium.dylib: Mach-O 64-bit dynamically linked shared library arm64
> file ./19/runtimes/osx-x64/native/libsodium.dylib
./19/runtimes/osx-x64/native/libsodium.dylib: Mach-O 64-bit dynamically linked shared library x86_64

In the later version, they became universal (fat) binaries:

> curl -fsSL -o 19.1.zip https://www.nuget.org/api/v2/package/libsodium/1.0.19.1 && unzip 19.1.zip -d 19.1
> file ./19.1/runtimes/osx-arm64/native/libsodium.dylib
./19.1/runtimes/osx-arm64/native/libsodium.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64:Mach-O 64-bit dynamically linked shared library arm64]
./19.1/runtimes/osx-arm64/native/libsodium.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
./19.1/runtimes/osx-arm64/native/libsodium.dylib (for architecture arm64):	Mach-O 64-bit dynamically linked shared library arm64
> file ./19.1/runtimes/osx-x64/native/libsodium.dylib
./19.1/runtimes/osx-x64/native/libsodium.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64:Mach-O 64-bit dynamically linked shared library arm64]
./19.1/runtimes/osx-x64/native/libsodium.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
./19.1/runtimes/osx-x64/native/libsodium.dylib (for architecture arm64):	Mach-O 64-bit dynamically linked shared library arm64

This supports @ektrah's spot-on sleuthing above.

I also think that the best avenue to figure out why published output differs would be to open an issue on the .NET SDK or runtime repos. The macOS .dylibs seem valid in both package versions, and the unexpected publish behavior likely requires .NET SDK expertise to diagnose.

[samuel-lucas6]

Thanks both, I'll open an issue or discussion in a .NET repo and link them here.