Enable use to the rebar feature to upload buffers to the device.

[mtavenrath]

Instead of copying host -> host staging -> device one can use the rebar feature to directly copy host -> device skipping the latency and 2nd memcpy which tripples memory bw consumption.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[mtavenrath]

Benchmark with Mistral-Nemo-Instruct-2407.Q5_K.gguf

| model                          |       size |     params | backend    | ngl |          nkvo | mmap |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---: | ------------: | ---------------: |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         pp512 |    431.11 ± 3.66 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         pp512 |    438.10 ± 0.52 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         pp512 |    434.48 ± 1.57 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         tg128 |     30.55 ± 0.75 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         tg128 |     31.73 ± 0.11 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         tg128 |     31.62 ± 0.13 |

rebar

| model                          |       size |     params | backend    | ngl |          nkvo | mmap |          test |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---: | ------------: | ---------------: |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         pp512 |    435.98 ± 1.63 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         pp512 |    433.19 ± 1.01 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         pp512 |    435.82 ± 0.89 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         tg128 |     35.07 ± 0.14 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         tg128 |     34.89 ± 0.20 |
| llama 13B Q5_K - Medium        |   8.12 GiB |    12.25 B | Vulkan     | 100 |             1 |    0 |         tg128 |     34.36 ± 0.19 |

[0cc4m]

@mtavenrath I'm sorry that it took me so long to get to the review.

Thank you for the contribution, this makes a significant difference in specific cases. Looks good to me.

[mtavenrath]

@0cc4m No worries, this was just the prelude to my current work with win32 iorings to get amazingly fast load times. On Windows I've been able to read ~45gb/s from 4xNVMe with a Win32 LVM raid into CPU memory.

The open question is if reading to GPU memory exposed through rebar will result in similar read performance. If so, the next question is how can we either expose CPU pointers of the Vulkan rebar tensors to llama.cpp? If this is not possible the question which arises next is ggml should be able to support file i/o to hit the fastest possible path.

[slaren]

You could probably make the buffer type a host buffer (return true from the is_host function of the buffer type interface), and set tensor::data to the CPU pointer. llama.cpp can load data directly into the tensor if it is allocated in a host buffer, and it also makes the tensor data directly accesible to the CPU backend without intermediate copies (but that's not always faster). You would probably need to make other changes to the Vulkan backend to find the device pointer where it is needed (which you can probably calculate from the CPU pointer).