I'm having "tensor buffer not set" while running cvector-generator:

  * frame #0: 0x0000000190862600 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x000000019089af70 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001907a7908 libsystem_c.dylib`abort + 128
    frame #3: 0x00000001000211f8 llama-cvector-generator`ggml_abort(file="ggml/src/ggml-backend.cpp", line=261, fmt=<unavailable>) at ggml.c:169:5 [opt]
    frame #4: 0x0000000100038104 llama-cvector-generator`ggml_backend_tensor_set(tensor=<unavailable>, data=<unavailable>, offset=<unavailable>, size=<unavailable>) at ggml-backend.cpp:261:5 [opt]
    frame #5: 0x00000001001d5d44 llama-cvector-generator`main at pca.hpp:321:9 [opt]
    frame #6: 0x00000001001d572c llama-cvector-generator`main(argc=<unavailable>, argv=<unavailable>) at cvector-generator.cpp:492:9 [opt]
    frame #7: 0x0000000190518274 dyld`start + 2840

The command that I used:

make llama-cvector-generator -j
./llama-cvector-generator -m ../models/meta-llama-3.1-8b-instruct-abliterated.Q4_K_M.gguf -ngl 99

This is likely caused by all the direct assignments of tensor->data with malloc. This not compatible with ggml-backend, the tensors need to be allocated into a buffer with ggml_backend_alloc_ctx_tensors or similar.

While trying to reproduce this I found other problems:

• covariance in run_single_pca is not big enough and causes a buffer overflow
• The tensors saved in the gguf do not have names, so it fails when they are added to the gguf file

The last one is especially puzzling to me.. was this tested at all?

@ngxson Oof, I'll have to investigate to find out what is going wrong, probably during the weekend, when computing the pca on float matrices during my tests this didn't happen.

@slaren I don't think there is any assignment directly to tensor->data with malloc on the pca.hpp file anymore, I've changed to use ggml_backend_tensor_set and ggml_backend_tensor_get, maybe is something related with this ? Btw I don't understand why the covariance matrix would not be big enough

The malloc are in cvector-generator.cpp. This is a possible way to solve that:

diff --git a/examples/cvector-generator/cvector-generator.cpp b/examples/cvector-generator/cvector-generator.cpp
index e7c924fb..17881c3a 100644
--- a/examples/cvector-generator/cvector-generator.cpp
+++ b/examples/cvector-generator/cvector-generator.cpp
@@ -2,6 +2,8 @@
 #include "common.h"
 #include "llama.h"
 #include "ggml.h"
+#include "ggml-cpp.h"
+#include "ggml-alloc.h"

 #include "mean.hpp"
 #include "pca.hpp"
@@ -193,6 +195,7 @@ struct train_context {
     // to easily re-alloc when concat v_diff, we temporary store v_diff in a vector instead of a tensor
     // v_diff_tmp will get converted unto v_diff later on
     std::vector<std::vector<uint8_t>> v_diff_tmp;
+    ggml_backend_buffer_ptr buffer;

     train_context(int n_embd_, int n_layers_) {
         n_embd = n_embd_;
@@ -207,9 +210,9 @@ struct train_context {
             std::vector<uint8_t> empty;
             v_diff_tmp.push_back(empty);
             auto t = ggml_new_tensor_1d(ctx_ggml, GGML_TYPE_F32, n_embd);
-            t->data = malloc(ggml_nbytes(t)); // TODO: get rid of malloc if possible
             v_final.push_back(t);
         }
+        buffer.reset(ggml_backend_alloc_ctx_tensors_from_buft(ctx_ggml, ggml_backend_cpu_buffer_type()));
     }

     // add new rows into existing tensor in v_diff_tmp

Btw I don't understand why the covariance matrix would not be big enough

I don't know what's the size supposed to be, but if you build with LLAMA_SANITIZE_ADDRESS=1 you should see the error. It may depend on the model, I used the very small stories260k.gguf from https://huggingface.co/ggml-org/models/tree/main/tinyllamas

Oooh the mallocs are in cvector-generator.cpp, seems like this file will need to be refactored then

Thanks! I'll test everything on the weekend and try to reproduce the problem with these models and investigate why the covariance matrix is overflowing to solve it aswell

Please note that some malloc was used in cvector-generator.cpp because some tensor size is unknown from the beginning (i.e. some depends on the number of input tokens)

There are other places that I simply use malloc to temporary store tensor data before writing it to GGUF.

Refactoring those would be nice.