Add ring buffer to store prev tokens in sampling

[kylo5aby]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[compilade]

Small question: how does this differ from std::queue or std::deque?

[kylo5aby]

Here I want to use a fixed capacity buffer to avoid resize or copy overhead, because the size of pre tokens for sampling is already known.

[compilade]

Right, std::deque can't reserve() like std::vector. This seems like a valid reason.

Might be worth it to write a (small) comment near ring_buffer to explain this.

[kylo5aby]

Resolved. Thanks the feedback!

[ggerganov]

I changed the base branch to gg/llama-refactor-sampling since it's better to merge this change together with the sampling refactoring

[ggerganov]

Let's remove these logs completely for now - will bring them back after the logger is reimplemented:

Suggested change

	LOG("last: %s\n", LOG_TOKENS_TOSTR_PRETTY(ctx, ctx_sampling->prev.to_vector()).c_str());
	//LOG("last: %s\n", LOG_TOKENS_TOSTR_PRETTY(ctx, ctx_sampling->prev.to_vector()).c_str());

[ggerganov]

Suggested change

	LOG("last: %s\n", LOG_TOKENS_TOSTR_PRETTY(ctx, ctx_sampling->prev.to_vector()).c_str());
	//LOG("last: %s\n", LOG_TOKENS_TOSTR_PRETTY(ctx, ctx_sampling->prev.to_vector()).c_str());

[ggerganov]

Should think of a way to avoid the to_vector() due to performance considerations

[kylo5aby]

Should think of a way to avoid the to_vector() due to performance considerations

I think one way to avoid vector copy is here we can pass penalty_prompt_tokens vector and start index to llama_sample_repetition_penalties, and then traverse penalty_last_n elements from the vector in it, which will avoid the copy. For example

void llama_sample_repetition_penalties(
            struct llama_context * ctx,
          llama_token_data_array * candidates,
                         // const llama_token * last_tokens,
                          const vector<llama_token>& penalty_tokens,
                          size_t start_index, // .size() - penalty_tokens_used_size or (prev.first +.size() - penalty_tokens_used_size) % prev.capacity if ring buffer
                          size_t   penalty_last_n,
                           float   penalty_repeat,
                           float   penalty_freq,
                           float   penalty_present);

what do you think?

[ggerganov]

After the sampling refactoring, the common/sampling.h/.cpp stuff will be moved to llama-sampling.cpp and the API call will become simply:

void llama_sampling_repetition_penalties(
           struct llama_sampling * ctx,
          llama_token_data_array * candidates);

All the penalty related information (together with ring buffer with the previous tokens) will be inside the llama_sampling object and we can handle it there. So for now, we can just resolve the conflict and merge and later I'll avoid the to_vector()