Request for guidance on retrieving input and output token counts

[MNIKIEMA]

Hi there,

Hi there,

Firstly, I'd like to thank you for developing and maintaining this library. It's a strong component for LLM, and I appreciate the effort that goes into it.

I'm currently working on a project where comprehensive token usage is crucial, specifically the number of input and output tokens when making requests to the GPT-3.5 Turbo model. However, I'm having trouble finding this information in the documentation or in the response object.

Here's a snippet of the code I'm using:

client = instructor.patch(OpenAI())
response = client.chat.completions.acreate(
    model="gpt-3.5-turbo-0613",
    response_model=PydanticModel,
    messages=[{"role": "user", "content": prompt}],
)

I'd like to know if there's a way to find out:

• The total number of input tokens processed in the request.
• The total number of output tokens generated in the response.

Understanding the number of tokens is critical to managing my application's quota and optimizing usage. If this feature is not available, could you consider it for future releases or suggest a possible workaround?

For context, in the OpenAI library, token counts can be retrieved as follows:

input_token_length = response.usage.prompt_tokens
input_token_length = response.usage.completion_tokens

However, it appears that the library does not support similar functionality, or it's not clearly documented. I'm seeking guidance on how I might achieve similar token tracking with instructor.

Thank you for your assistance, and I look forward to hearing from you.

[jxnl]

https://jxnl.github.io/instructor/concepts/raw_response/?h=completion+tokens

[jxnl]

#172

[jxnl]

can you let me know where you looked so i can make this more obvious?

[MNIKIEMA]

Thank you for your feedback. I've reviewed the Cookbook and the Concepts section; however, my focus was on identifying information specifically related to token usage.
It would be beneficial to have a dedicated section where users can easily find guidance on configuring parameters such as max_new_token, temperature, and other settings related to the OpenAI API.

[jxnl]

that should be part of the normal sdk

[jxnl]

I'll add a section called token usage in concepts