Skip to content

Commit

Permalink
fix: add max_tokens when using vision model (#4157)
Browse files Browse the repository at this point in the history
  • Loading branch information
fred-bf authored Feb 27, 2024
1 parent 44a5127 commit 08fa227
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions app/client/platforms/openai.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,16 @@ export class ChatGPTApi implements LLMApi {
// Please do not ask me why not send max_tokens, no reason, this param is just shit, I dont want to explain anymore.
};

// add max_tokens to vision model
if (visionModel) {
Object.defineProperty(requestPayload, "max_tokens", {
enumerable: true,
configurable: true,
writable: true,
value: Math.max(modelConfig.max_tokens, 4096),
});
}

console.log("[Request] openai payload: ", requestPayload);

const shouldStream = !!options.config.stream;
Expand Down

5 comments on commit 08fa227

@ficapy
Copy link

@ficapy ficapy commented on 08fa227 Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fred-bf This submission triggers the use of models outside the DEFAULT_MODELS list, automatically adding a max_tokens = 4096 parameter, which leads to errors
I used together.ai + Qwen/Qwen1.5-72B-Chat. The service provider returned the following error:

{
  "error": {
    "message": "Input validation error: `inputs` tokens + `max_new_tokens` must be <= 4097. Given: 115 `inputs` tokens and 4096 `max_new_tokens`",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": null
  }
}

@H0llyW00dzZ
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fred-bf This submission triggers the use of models outside the DEFAULT_MODELS list, automatically adding a max_tokens = 4096 parameter, which leads to errors I used together.ai + Qwen/Qwen1.5-72B-Chat. The service provider returned the following error:

{
  "error": {
    "message": "Input validation error: `inputs` tokens + `max_new_tokens` must be <= 4097. Given: 115 `inputs` tokens and 4096 `max_new_tokens`",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": null
  }
}

I've been knowing that an error will occur due to the setting value: Math.max(modelConfig.max_tokens, 4096).
Ideally, Math.max should not be set, as the maximum error handling is already managed by the provider.

@fred-bf
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@H0llyW00dzZ You are right, just created a hot fix for this 43e5dc2

@fred-bf
Copy link
Contributor Author

@fred-bf fred-bf commented on 08fa227 Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ficapy the issue have been fixed in the latest commit, the original way to detect vision model is regarding all unrecognized model as vision model https://github.com/ChatGPTNextWeb/ChatGPT-Next-Web/blob/08fa22749aea8f497811f684bd9c7ef68d698666/app/utils.ts#L297

@H0llyW00dzZ
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@H0llyW00dzZ You are right, just created a hot fix for this 43e5dc2

Now, it is better and more straightforward.

Please sign in to comment.