llama : refactor sampling v2

[ggerganov]

alt: #8643

ref: #5214

Overview

• Replace llama_sampling_ and llama_grammar_ with new llama_sampler_
• Overhaul common: struct llama_sampling_context in common: struct gpt_sampler
• Support for user-defined samplers via struct llama_sampler_i interface

API Changes

• Add struct llama_sampler and struct llama_sampler_i
• Add llama_sampler_ API
• Add llama_sampler_chain_ API for chaining multiple samplers
• Remove LLAMA_API_INTERNAL
• Remove Classifier-Free Guidance related stuff
• Remove Prompt Penalty support
• Add llama_perf_ API and remove old llama_print_timings and llama_reset_timings

Implementation details

• Move common/grammar-parser in src/llama-grammar
• The llama_context no longer comes with a built-in sampling context. The user code is responsible for creating, using, saving and loading the llama_sampler objects as needed. As a consequence, the llama_state no longer serializes the RNG
• The grammar code has been refactored, hopefully it is a bit easier to read. No functional changes.
• The samplers implemented in llama-sampling.cpp can be used as examples for implementing custom samplers in user code

Example

Comparison of user sampling code before and after:

• before

// decoding loop:
auto   n_vocab = llama_n_vocab(model);
auto * logits  = llama_get_logits_ith(ctx, i_batch[i]);

std::vector<llama_token_data> candidates;
candidates.reserve(n_vocab);

for (llama_token token_id = 0; token_id < n_vocab; token_id++) {
    candidates.emplace_back(llama_token_data{ token_id, logits[token_id], 0.0f });
}

llama_token_data_array candidates_p = { candidates.data(), candidates.size(), false };

llama_sample_top_k(ctx, &candidates_p, top_k, 1);
llama_sample_top_p(ctx, &candidates_p, top_p, 1);
llama_sample_temp (ctx, &candidates_p, temp);

const llama_token new_token_id = llama_sample_token(ctx, &candidates_p);

• after

// prepare the sampling chain at the start
auto sparams = llama_sampler_chain_default_params();

llama_sampler * smpl = llama_sampler_chain_init(sparams);

llama_sampler_chain_add(smpl, llama_sampler_init_top_k(top_k));
llama_sampler_chain_add(smpl, llama_sampler_init_top_p(top_p, 1));
llama_sampler_chain_add(smpl, llama_sampler_init_temp (temp));
llama_sampler_chain_add(smpl, llama_sampler_init_dist (seed));

...

// decoding loop:
const llama_token new_token_id = llama_sampler_sample(smpl, ctx, i_batch[i]);

Future plan and ideas

• Attach samplers to the decoding graphs for better performance (e.g. llama_decode_with_sampler())
• Extend llama_decode_ API to support multiple decoding runs (e.g. llama_decode_n())
• Existing samplers implementation in llama-sampling.cpp could be split into separate source files
• Expose struct llama_vocab through the public API and change calls that currently use struct llama_model to use it when appropriate
• Deduplicate the ring_buffer code by implementing ggml_ring_buffer for fixed-size objects
• Measure and report the performance of the grammar
• Simplify llama_token_data llama : refactor sampling v2 #9294 (review)

---

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[ExtReMLapin]

Not sure it's the right place or time to talk about this but in another issue a guy had the idea of, "if the grammar says the character/word can only be "xxx" and nothing else, don't bother asking the LLM what to say for the X next tokens".

As there is a refactoring going on, maybe it's the right time to implement it.

[ggerganov]

It's not in the scope of this change, but also the grammar never fits only one token. For example, all three tokens "x", "xx" and "xxx" would fit the grammar in that case. One way would be to use the longest token. Another way would be to tokenize "xxx" and use the resulting tokens. Not sure

[ggerganov]

This is getting close to ready. Later today will add detailed description of the changes, some comments in the code and do a bit more testing.

@slaren PTAL - any comments and suggestions are welcome.

[slaren]

I don't know what's the history of the min_keep parameter in all of these samplers, from what I can tell parameter is not used in the examples except by the server, but it seems very suspect to me.

Edit: looks like it has been there since the beginning (#1126), and there was never any explanation of why it is needed.

[ggerganov]

Removed min_keep from the top_k sampler as it didn't make sense. For the p-based samplers, I think it makes sense to guarantee minimum number of candidate results, regardless of the p value.

[ggerganov]

Thanks for the review. I got side tracked a bit with a bug in the speculative example (should be fixed now). Will apply the review tomorrow and prepare this for merging.

[clarismiranda]

Hello! Why does llama_grammar_accept return the stacks? It was previously passed by reference

[ggerganov]

Thanks for noticing. I changed it because I thought it improves the signature of the function, but I missed that it would lead to extra memory allocations. So I restored the original signature. f9762c6

Edit: though after one more look, I think it does not matter since we move stacks_new either way.

[slaren]

Looks good.

In the future we should probably simplify llama_token_data (or remove it entirely) to keep only one value per token and add a flag to llama_token_data_array to indicate whether the current values are probabilities (ie. normalized to sum 1) or not, so that samplers that can only operate with probabilities can know if they need to call softmax. Having two values per token is very confusing to me because some samplers operate on one or other, and this can lead to situations where a sampler modifies the probabilities, and the next one calls softmax which discards all the changes to the probabilities and computes them from the logits again. I cannot tell if there are already situations like that which end with some samplers that operate on probabilities being ignored.

[mudler]

Somehow after this change I see breakage in llava sampling - I'm still OOO so I didn't deep dive yet, see mudler/LocalAI#3497 for reference on how it breaks in LocalAI.

Would be really appreciated if anyone have an idea/pointers of what's going on ! Thank you!

10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stderr /home/mudler/_git/LocalAI/backend/cpp/llama-avx2/llama.cpp/ggml/src/ggml.c:13835: GGML_ASSERT(i01 >= 0 &
& i01 < ne01) failed


10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout [Thread debugging using libthread_db enabled]
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout Using host libthread_db library "/lib64/libthread_db.so.1".
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout 0x00007f989b8e94a3 in ?? () from /lib64/libgomp.so.1
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #0  0x00007f989b8e94a3 in ?? () from /lib64/libgomp.so.1
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #1  0x00000000008222e5 in ggml_graph_compute_thread.isra ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #2  0x00007f989b8dcd16 in GOMP_parallel () from /lib64/libgomp.so.1
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #3  0x0000000000825a2a in ggml_graph_compute ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #4  0x0000000000834010 in ggml_backend_cpu_graph_compute ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #5  0x000000000083784c in ggml_backend_graph_compute ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #6  0x0000000000652b63 in clip_image_batch_encode.constprop ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #7  0x0000000000653553 in clip_image_encode ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #8  0x0000000000657ac8 in llava_image_embed_make_with_clip_img ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #9  0x00000000004e2c09 in llama_server_context::update_slots() [clone .isra.0] ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #10 0x00000000004d7629 in llama_server_queue::start_loop() ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout #11 0x000000000048b040 in main ()
10:25PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:42747): stdout [Inferior 1 (process 13029) detached]

[ggerganov]

Which commit are you using? I think you need to update to 1b28061

[mudler]

Which commit are you using? I think you need to update to 1b28061

Thanks for the quick reply! I was at daa9623, I'll try with that and let you know ASAP

[mudler]

mmh, tried with the latest commit (e6b7801) but still crashing with:

6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stderr /home/mudler/_git/LocalAI/backend/cpp/llama-avx2/llama.cpp/ggml/src/ggml.c:13853: GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
...
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout [Thread debugging using libthread_db enabled]
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout Using host libthread_db library "/lib64/libthread_db.so.1".
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout 0x00007fd8a45ee4a3 in ?? () from /lib64/libgomp.so.1
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #0  0x00007fd8a45ee4a3 in ?? () from /lib64/libgomp.so.1
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #1  0x00000000007dd4b5 in ggml_graph_compute_thread.isra ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #2  0x00007fd8a45e1d16 in GOMP_parallel () from /lib64/libgomp.so.1
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #3  0x00000000007e0cca in ggml_graph_compute ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #4  0x00000000007ef340 in ggml_backend_cpu_graph_compute ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #5  0x00000000007f2b7c in ggml_backend_graph_compute ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #6  0x000000000060d8b3 in clip_image_batch_encode.constprop ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #7  0x000000000060e2a3 in clip_image_encode ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #8  0x0000000000612818 in llava_image_embed_make_with_clip_img ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #9  0x00000000004dd269 in llama_server_context::update_slots() [clone .isra.0] ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #10 0x00000000004d1ce9 in llama_server_queue::start_loop() ()
6:24PM DBG GRPC(moondream2-text-model-f16.gguf-127.0.0.1:44339): stdout #11 0x0000000000486a10 in main ()

[slaren]

That's not related to the sampling changes, the only difference is that get_rows operations are bound-checked in all builds now, while previously it was only checked in debug builds. The clip implementation is broken and needs to be fixed: #9066 (comment)

[mudler]

That's not related to the sampling changes, the only difference is that get_rows operations are bound-checked in all builds now, while previously it was only checked in debug builds. The clip implementation is broken and needs to be fixed: #9066 (comment)

Thanks for that bit, totally missed it. What it's weird is that now for me it's a 100% hit since I started pinning new version of llama.cpp - as I have test suites running vision tests this never popped up until now. It's not sporadic at all, but really consistent and can't get a single time passing

Commit still working here: 815b1fb
Commit which is not working: e6b7801 (which includes #9082 )

[slaren]

You would need to run the test suite with a debug build to be able to hit the assert. If you want the previous behavior you can revert #9354 in your build, but that still does not make it any less broken, it just hides the issue.

[mudler]

You would need to run the test suite with a debug build to be able to hit the assert. If you want the previous behavior you can revert #9354 in your build, but that still does not make it any less broken, it just hides the issue.

Thanks for the hints @slaren - I actually tried to comment the assert at all as well to double check - but as you suggested it "hides" it, and crashes in the same way.

another datapoint from my side - seems the suggestion in the comment, to edit

diff --git a/examples/llava/clip.cpp b/examples/llava/clip.cpp
index 342042ff..224db9b5 100644
--- a/examples/llava/clip.cpp
+++ b/examples/llava/clip.cpp
@@ -2419,7 +2419,7 @@ bool clip_image_batch_encode(clip_ctx * ctx, const int n_threads, const clip_ima
             struct ggml_tensor * patches = ggml_graph_get_tensor(gf, "patches");
             int* patches_data = (int*)malloc(ggml_nbytes(patches));
             for (int i = 0; i < num_patches; i++) {
-                patches_data[i] = i + 1;
+                patches_data[i] = i;
             }
             ggml_backend_tensor_set(patches, patches_data, 0, ggml_nbytes(patches));
             free(patches_data);

actually "fixes" the issue here - probably it's not ideal, but at least seems indeed something is off in the clip implementation when loading images. Sorry for making noise here - is there already an issue open for this issue or shall I open one? I can't find one for this specific issue

[slaren]

I don't think there is an open issue about this currently, I know that it was briefly discussed in #9066, but that one is already closed.

[ngxson]

Maybe worth to notice: signature of llama_sampling_sample is slightly different from the new gpt_sampler_sample:

llama_token llama_sampling_sample(
        struct llama_sampling_context * ctx_sampling,
        struct llama_context * ctx_main,
        struct llama_context * ctx_cfg,
        int idx = -1);

llama_token gpt_sampler_sample(
    struct gpt_sampler * gsmpl,
    struct llama_context * ctx,
    int idx,
    bool grammar_first = false);

Compiler may not though any error or warning because values set to llama_sampling_sample can also be cast to new values required by gpt_sampler_sample

[ggerganov]

Compiler may not though any error or warning because values set to llama_sampling_sample can also be cast to new values required by gpt_sampler_sample

Hm, it will generate an error. Don't think there is an issue.

[zhaoyinglia]

@ggerganov Hi, I'm a bit confused. Why was Classifier-Free Guidance removed? Are there any issues with it?

[ggerganov]

The implementation does not need to be part of libllama because it works on the logits and is trivial to implement in user code. I removed it also from the examples, because it was much simpler this way and refactoring the sampling was with higher priority than keeping CFG functional. We can reintroduce this functionality in one of the examples or in a new dedicated example if there is interest. PRs welcome.