[Bug]: Basten model integration not working as expected

[usersina]

What happened?

I deployed a model to Basten and added it with its id in my litellm_config.yaml.
It might be that the Baseten integration defaults to stream even though the option is disabled.

model_list:
  - model_name: open-mixtral-8x7b
      litellm_params:
      model: baseten/zq8z2n5q
      api_key: os.environ/BASTEN_API_KEY

Relevant log output

16:40:58 - LiteLLM:DEBUG: utils.py:283 - RAW RESPONSE:
{"model_id":"zq8z2n5q","model_version_id":"qvrkrrw","model_output":"\n    -Unspecified Activity: Beach / Food Markets\n    -Unspecified Place: None\n\nI will be happy to help you plan a 5-day travel itinerary to Granada, Spain, with a focus"}


16:40:58 - LiteLLM:DEBUG: main.py:5259 - raw model_response: {"model_id":"zq8z2n5q","model_version_id":"qvrkrrw","model_output":"\n    -Unspecified Activity: Beach / Food Markets\n    -Unspecified Place: None\n\nI will be happy to help you plan a 5-day travel itinerary to Granada, Spain, with a focus"}
16:40:58 - LiteLLM:DEBUG: cost_calculator.py:576 - completion_response response ms: None 
16:40:58 - LiteLLM:DEBUG: utils.py:4294 - model=baseten/zq8z2n5q, custom_llm_provider=baseten has no input_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM:DEBUG: utils.py:4306 - model=baseten/zq8z2n5q, custom_llm_provider=baseten has no output_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM:DEBUG: utils.py:283 - Returned custom cost for model=baseten/zq8z2n5q - prompt_tokens_cost_usd_dollar: 0, completion_tokens_cost_usd_dollar: 0
16:40:58 - LiteLLM:DEBUG: utils.py:283 - Async Wrapper: Completed Call, calling async_success_handler: <bound method Logging.async_success_handler of <litellm.litellm_core_utils.litellm_logging.Logging object at 0x7f102f4faea0>>
16:40:58 - LiteLLM Router:INFO: router.py:942 - litellm.acompletion(model=baseten/zq8z2n5q) 200 OK
16:40:58 - LiteLLM:DEBUG: utils.py:4294 - model=baseten/zq8z2n5q, custom_llm_provider=baseten has no input_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM:DEBUG: utils.py:4306 - model=baseten/zq8z2n5q, custom_llm_provider=baseten has no output_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM Router:DEBUG: router.py:2693 - Async Response: ModelResponse(id='chatcmpl-5b9ae30a-5b50-41c8-80d1-b555f2869c8e', created=1737132058, model='zq8z2n5q', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='stop', index=0, message=Message(content='\n    -Unspecified Activity: Beach / Food Markets\n    -Unspecified Place: None\n\nI will be happy to help you plan a 5-day travel itinerary to Granada, Spain, with a focus', role='assistant', tool_calls=None, function_call=None))], usage=Usage(completion_tokens=43, prompt_tokens=167, total_tokens=210, completion_tokens_details=None, prompt_tokens_details=None))
16:40:58 - LiteLLM:DEBUG: utils.py:283 - Logging Details LiteLLM-Async Success Call, cache_hit=None
16:40:58 - LiteLLM:DEBUG: cost_calculator.py:576 - completion_response response ms: 1303.63 
16:40:58 - LiteLLM:DEBUG: utils.py:4294 - model=baseten/zq8z2n5q, custom_llm_provider=baseten has no input_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM:DEBUG: utils.py:4306 - model=baseten/zq8z2n5q, custom_llm_provider=baseten has no output_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM:DEBUG: utils.py:283 - Returned custom cost for model=baseten/zq8z2n5q - prompt_tokens_cost_usd_dollar: 0, completion_tokens_cost_usd_dollar: 0
16:40:58 - LiteLLM:DEBUG: utils.py:4294 - model=zq8z2n5q, custom_llm_provider=baseten has no input_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM:DEBUG: utils.py:4306 - model=zq8z2n5q, custom_llm_provider=baseten has no output_cost_per_token in model_cost_map. Defaulting to 0.
16:40:58 - LiteLLM Proxy:DEBUG: proxy_track_cost_callback.py:30 - INSIDE _PROXY_track_cost_callback
16:40:58 - LiteLLM Proxy:DEBUG: proxy_track_cost_callback.py:32 - kwargs stream: False + complete streaming response: None
16:40:58 - LiteLLM:DEBUG: litellm_logging.py:921 - Logging Details LiteLLM-Success Call: Cache_hit=None
16:40:58 - LiteLLM Proxy:DEBUG: proxy_track_cost_callback.py:61 - user_api_key 581d30b4caecfdbad68803e06cde0f826b06850314fbadc988d9ad6d0d4d3298, prisma_client: <litellm.proxy.utils.PrismaClient object at 0x7f102f7acd70>
16:40:58 - LiteLLM:DEBUG: cost_calculator.py:576 - completion_response response ms: 1303.63

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

ghcr.io/berriai/litellm-database:main-v1.56.10

Twitter / LinkedIn details

No response