Skip to content

Commit

Permalink
partners: Fixed the procedure of initializing pad_token_id
Browse files Browse the repository at this point in the history
Add to check pad_token_id and eos_token_id of model config.
It seems that this is the same bug as the HuggingFace TGI bug.
  • Loading branch information
tishizaki committed Jan 30, 2025
1 parent 22219ee commit e8a8198
Showing 1 changed file with 10 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,16 @@ def from_model_id(
model = model_cls.from_pretrained(model_id, **_model_kwargs)

if tokenizer.pad_token is None:
tokenizer.pad_token_id = model.config.eos_token_id
if model.config.pad_token_id is not None:
tokenizer.pad_token_id = model.config.pad_token_id
elif model.config.eos_token_id is not None and isinstance(
model.config.eos_token_id, int
):
tokenizer.pad_token_id = model.config.eos_token_id
elif tokenizer.eos_token_id is not None:
tokenizer.pad_token_id = tokenizer.eos_token_id
else:
tokenizer.add_special_tokens({"pad_token": "[PAD]"})

if (
(
Expand Down

0 comments on commit e8a8198

Please sign in to comment.