Skip to content

Commit

Permalink
Fix fastchat top_k (intel-analytics#10560)
Browse files Browse the repository at this point in the history
* fix -1 top_k

* fix

* done
  • Loading branch information
gc-fu authored Mar 27, 2024
1 parent fc8c790 commit 04baac5
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion python/llm/src/ipex_llm/serving/fastchat/ipex_llm_worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,9 @@ def generate_stream_gate(self, params):
temperature = float(params.get("temperature", 1.0))
repetition_penalty = float(params.get("repetition_penalty", 1.0))
top_p = float(params.get("top_p", 1.0))
top_k = int(params.get("top_k", 0)) # 0 means disable
top_k = int(params.get("top_k", 1))
if top_k == -1:
top_k = 1
max_new_tokens = int(params.get("max_new_tokens", 256))
echo = bool(params.get("echo", True))
stop_str = params.get("stop", None)
Expand Down

0 comments on commit 04baac5

Please sign in to comment.