[Bug] Max tokens parameter is being incorrect set

[yuzukumo]

Bug Description

Max toekns has been regulated to max 512,000 while the models support more than this limit. For example, gemini-1.5-pro-latest supports a max tokens of 1,048,576.

Steps to Reproduce

Adjust the max tokens parameter.

Expected Behavior

Max 512,000.

Screenshots

No response

Deployment Method

• Docker
• Vercel
• Server

Desktop OS

Windows 11

Desktop Browser

Edge

Desktop Browser Version

124

Smartphone Device

No response

Smartphone OS

No response

Smartphone Browser

No response

Smartphone Browser Version

No response

Additional Logs

No response

[yuzukumo]

A PR has been submitted to fix it: #4597

[Algorithm5838]

The max_tokens setting is often misunderstood, and its description in this project's settings is incorrect. To clarify, max_tokens does not refer to the LLM's context, which is the combination of input and output tokens. Instead, max_tokens limits the number of output tokens generated by the LLM.

To illustrate this, consider the gpt-4-turbo model, which has a context size of 128,000 tokens. However, its output is capped at 4,096 tokens. To obtain the maximum output, you should set max_tokens to 4,096. If you set it to 128,000, you'll encounter an error, as you won't leave sufficient tokens for the input.

Note that max_tokens is disabled by default in this project, but enabled for vision models with a setting of 4,000. If you're using a custom fork, be sure to enable max_tokens accordingly.

[yuzukumo]

Really thx for the clarification!