Consistency changes

[ericcurtin]

Ensure llama.cpp version is the same accross containers. Removing some duplicate actions, etc.

[ericcurtin]

PTAL @bmahabirbu I didn't test the cuda changes FWIW, don't have the hardware

[bmahabirbu]

I tested it and it crashed before loading the model with a segmentation fault. I originally thought it had something to do with the linker not updating the correct libggml.so file but its just actually do to the second cmakle --install line for whisper. It renames some llama stuff despite the mv commands causing issues

[bmahabirbu]

Line 34

cmake --install build && \

Comment this out and we're good to go

[ericcurtin]

This line installs the libwhisper stuff:

$ ls /usr/lib64/libw*
/usr/lib64/libwhisper.so  /usr/lib64/libwhisper.so.1  /usr/lib64/libwhisper.so.1.7.1

[ericcurtin]

I had to add this (nvidia-driver-cuda-libs) for this lib:

/lib64/libcuda.so.1

$ ldd /usr/bin/llama-cli
	linux-vdso.so.1 (0x00007f219dc64000)
	libllama.so => /lib64/libllama.so (0x00007f219dae4000)
	libggml.so => /lib64/libggml.so (0x00007f2188000000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f2187c00000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f219da09000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f219d9ee000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2187800000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f219dc66000)
	libcudart.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12 (0x00007f2187400000)
	libcublas.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.12 (0x00007f2180a00000)
	libcublasLt.so.12 => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.12 (0x00007f215f000000)
	libcuda.so.1 => /lib64/libcuda.so.1 (0x00007f215c000000)
	libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f219d9a5000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f219d99e000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f219d999000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f219d994000)

which makes me wonder, how was this working before without explicitly installing that.

[bmahabirbu]

Its due to cuda container toolkit passing the library from the nvidia driver on the host to the container. If you install the driver inside the container you'll get an error like this

Error: OCI runtime error: error executing hook `/usr/bin/nvidia-container-toolkit`

[ericcurtin]

Should we remove nvidia-driver-cuda-libs? Is it breaking nvidia-container-toolkit?

[ericcurtin]

I wonder can we CI some of this stuff even without Nvidia hardware in the CI runners

[ericcurtin]

I tested it and it crashed before loading the model with a segmentation fault. I originally thought it had something to do with the linker not updating the correct libggml.so file but its just actually do to the second cmakle --install line for whisper. It renames some llama stuff despite the mv commands causing issues

Yes I see what you mean, we have two different versions of /usr/lib64/libggml.so clashing

[ericcurtin]

We must somehow align them to the same version...

[ericcurtin]

We have at least three options to solve this I can think of, 1. split llama.cpp and whisper.cpp into two containerfiles

Or...

2. Ensure at build-time of the container images that llama.cpp and whisper.cpp versions are ABI compatible...

Or...

3. Statically link libggml.so into the whisper binaries

[ericcurtin]

I fixed it by building whisper with -DBUILD_SHARED_LIBS=NO , so using static linking for whisper.cpp (but not llama.cpp) seems like an ok option for now.

[bmahabirbu]

I fixed it by building whisper with -DBUILD_SHARED_LIBS=NO , so using static linking for whisper.cpp (but not llama.cpp) seems like an ok option for now.

Nice, I'll give it a test run as well

[bmahabirbu]

I fixed it by building whisper with -DBUILD_SHARED_LIBS=NO , so using static linking for whisper.cpp (but not llama.cpp) seems like an ok option for now.

Nice, I'll give it a test run as well

Confirmed working! (Had to remove the dnf install nvidia-driver-cuda-libs but that's just due to me setting my environment up with cuda container toolkit)

[ericcurtin]

I fixed it by building whisper with -DBUILD_SHARED_LIBS=NO , so using static linking for whisper.cpp (but not llama.cpp) seems like an ok option for now.

Nice, I'll give it a test run as well

Confirmed working! (Had to remove the dnf install nvidia-driver-cuda-libs but that's just due to me setting my environment up with cuda container toolkit)

Well lets remove it if that's the case, makes me wonder if somebody could run this without the nvidia-container-toolkit... But anyway don't want to break a working setup

[bmahabirbu]

I fixed it by building whisper with -DBUILD_SHARED_LIBS=NO , so using static linking for whisper.cpp (but not llama.cpp) seems like an ok option for now.

Nice, I'll give it a test run as well

Confirmed working! (Had to remove the dnf install nvidia-driver-cuda-libs but that's just due to me setting my environment up with cuda container toolkit)

Well lets remove it if that's the case, makes me wonder if somebody could run this without the nvidia-container-toolkit... But anyway don't want to break a working setup

Sounds good to me! I'll investigate some more to see if there is a way to run it without the container toolkit

[ericcurtin]

PTAL @rhatdan this is ready. A lot of this PR is bringing consistency to build and install approaches in the various container images.

One thing this fixes in llama.cpp and whisper.cpp is that they use different versions of libggml and we were getting mismatching ABI's sometimes if in a function call was in one but not the other. The second installed version of libggml would overwrite the first.

We fixes this by using statically linking in libggml in the two whisper.cpp binaries so llama.cpp can use different versions of libggml like upstream llama.cpp/whisper.cpp intends.