Skip to content

Commit

Permalink
fragmentation calculation fix
Browse files Browse the repository at this point in the history
  • Loading branch information
Xarbirus committed Mar 13, 2024
1 parent 283ab2f commit a71b695
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion llama.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8898,7 +8898,7 @@ static int llama_decode_internal(

// decide if we need to defrag the kv cache
if (cparams.defrag_thold >= 0.0f) {
const float fragmentation = kv_self.n >= 128 ? 1.0f - float(kv_self.used + n_tokens)/float(kv_self.n) : 0.0f;
const float fragmentation = kv_self.n >= 128 ? 1.0f - float(kv_self.used)/float(kv_self.n) : 0.0f;

// queue defragmentation for next llama_kv_cache_update
if (fragmentation > cparams.defrag_thold) {
Expand Down

0 comments on commit a71b695

Please sign in to comment.