Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ log all errored requests #30

Merged
merged 3 commits into from
May 20, 2024
Merged

✨ log all errored requests #30

merged 3 commits into from
May 20, 2024

Conversation

joerunde
Copy link
Collaborator

Drafting some changes to log all errors during validation or generation for a request like TGIS does. The goal is to get something like this:

2024-05-17T19:24:20.591834Z ERROR generate{input=["Hallo, wie heißt du? ..."] prefix_id=None correlation_id="<none>" input_bytes=[22] params=Some(Parameters { method: Sample, sampling: Some(SamplingParameters { temperature: 0.0, top_k: 3, top_p: 0.0, typical_p: 0.0, seed: None }), stopping: Some(StoppingCriteria { max_new_tokens: 20, min_new_tokens: 16, time_limit_millis: 0, stop_sequences: ["Peter ", "Timothy ", "joseph", "Corinthians"], include_stop_sequence: None }), response: Some(ResponseOptions { input_text: false, generated_tokens: false, input_tokens: false, token_logprobs: false, token_ranks: false, top_n_tokens: 0 }), decoding: Some(DecodingParameters { repetition_penalty: 2.0, length_penalty: Some(LengthPenalty { start_index: 0, decay_factor: 25.0 }) }), truncate_input_tokens: 0 })}: text_generation_router::grpc_server: src/grpc_server.rs:429: length_penalty must be >= 1.0 and <= 10.0

though probably without the file name/line number stuff (text_generation_router::grpc_server: src/grpc_server.rs:429 in the example above)

joerunde added 2 commits May 17, 2024 14:19
@joerunde joerunde marked this pull request as ready for review May 17, 2024 22:47
Copy link
Contributor

@njhill njhill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joerunde! only minor comments.

It feels like it would be better for the response logging to happen in one place (whether success or error), not sure how realistic that is but in any case can save for possible later refactor.

@joerunde
Copy link
Collaborator Author

Thanks for the review @njhill, everything looks like it still works!

Copy link
Contributor

@njhill njhill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joerunde!

@joerunde joerunde merged commit 066041a into main May 20, 2024
14 checks passed
@joerunde joerunde deleted the log-errors branch May 20, 2024 17:58
joerunde added a commit that referenced this pull request May 20, 2024
This PR logs all errors during validation or generation
for a request like TGIS does. 

Signed-off-by: Joe Runde <[email protected]>
tdoublep pushed a commit that referenced this pull request Jan 20, 2025
For each request is checked if the prompt length exceeds the largest
target padding length as specified for the supported shapes. It that
happens to be true, then the request is ignored and a corresponding
message is output by the server.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants