Usage of openai configuration throws exception Completions.create() got an unexpected keyword argument 'model_id'

[jwmatthews]

Seeing an issue of: Completions.create() got an unexpected keyword argument 'model_id', when I attempt to run with an OpenAI config.

I believe I see the issue and described the problem in this comment: #394 (comment)

We are merging values from a config file built inside of the container AND the specified config to run with, this creates an incompatibility when using a provider that is NOT using the keyword model_id

I used the example we have at https://github.com/konveyor/kai/blob/main/build/config_example.toml as a starting point and slightly tweaked the example for gpt-4o.

I am testing from a podman compose up flow, I built the latest code as of 9/26 into a tag of 'local'.
I am running the Kai server as: TAG="local" podman compose up

Next, I want to use a custom config, so I followed the pattern of copying the config_example.toml in the 'build' directory

$ cd build
$ cp config_example.toml

Then I edited the 'config.toml' to comment out the mixtral section and then make the below config active

$ cat build/config.toml 

...
...
[models]
provider = "ChatOpenAI"

[models.args]
model = "gpt-4o"

Below is what I see when I attempt to generate a fix for a file
Snippet of console output

kai_1               | INFO - 2024-09-27 12:03:22,555 - kai.routes.get_incident_solutions_for_file - [get_incident_solutions_for_file.py:38   - post_get_incident_solutions_for_file()] - START - App: 'coolstore', File: 'src/main/java/com/redhat/coolstore/service/InventoryNotificationMDB.java' with 2 incidents'
kai_1               | INFO - 2024-09-27 12:03:22,678 - kai.service.kai_application.kai_application - [  kai_application.py:133  - get_incident_solutions_for_file()] - Processing incident batch 1/1 with 2 incident(s) for src/main/java/com/redhat/coolstore/service/InventoryNotificationMDB.java
kai_1               | WARNING - 2024-09-27 12:03:22,717 - kai.service.kai_application.kai_application - [  kai_application.py:214  - get_incident_solutions_for_file()] - Request to model failed for batch 1/1 for src/main/java/com/redhat/coolstore/service/InventoryNotificationMDB.java with exception Completions.create() got an unexpected keyword argument 'model_id', retrying in 10.0s

[jwmatthews]

I next ran with make run-server and using config of below with success

$ cat kai/config.toml
# Default configuration file for Kai. For a better understanding of the
# configuration options, please refer to `build/example_config.toml`

log_level = "info"
file_log_level = "debug"
log_dir = "$pwd/logs"
demo_mode = false
trace_enabled = true

solution_consumers = ["diff_only", "llm_summary"]

[incident_store]
solution_detectors = "naive"
solution_producers = "text_only"

[incident_store.args]
provider = "postgresql"
host = "127.0.0.1"
database = "kai"
user = "kai"
password = "dog8code"

[models]
#provider = "ChatIBMGenAI"
provider = "ChatOpenAI"

[models.args]
model = "gpt-4o"
#model_id = "meta-llama/llama-3-70b-instruct"
#parameters.max_new_tokens = 2048

Below is sample of console output as I generated a fix.

INFO - 2024-09-27 08:26:09,113 - kai.server - [           server.py:58   -                  app()] - Kai server is ready to receive requests.
INFO - 2024-09-27 08:26:09,129 - kai.service.kai_application.kai_application - [  kai_application.py:63   -             __init__()] - Selected provider: ChatOpenAI
INFO - 2024-09-27 08:26:09,130 - kai.service.kai_application.kai_application - [  kai_application.py:64   -             __init__()] - Selected model: gpt-4o
INFO - 2024-09-27 08:26:09,166 - kai.service.kai_application.kai_application - [  kai_application.py:84   -             __init__()] - Selected incident store: postgresql
INFO - 2024-09-27 08:26:09,166 - kai.server - [           server.py:58   -                  app()] - Kai server is ready to receive requests.
INFO - 2024-09-27 08:26:30,223 - kai.routes.get_incident_solutions_for_file - [get_incident_solutions_for_file.py:38   - post_get_incident_solutions_for_file()] - START - App: 'coolstore', File: 'src/main/java/com/redhat/coolstore/service/InventoryNotificationMDB.java' with 2 incidents'
INFO - 2024-09-27 08:26:30,403 - kai.service.kai_application.kai_application - [  kai_application.py:133  - get_incident_solutions_for_file()] - Processing incident batch 1/1 with 2 incident(s) for src/main/java/com/redhat/coolstore/service/InventoryNotificationMDB.java
WARNING - 2024-09-27 08:26:48,002 - kai.service.kai_application.kai_application - [  kai_application.py:189  - get_incident_solutions_for_file()] - Key does not exist in the dictionary: 'token_usage'
INFO - 2024-09-27 08:26:48,024 - kai.routes.get_incident_solutions_for_file - [get_incident_solutions_for_file.py:73   - post_get_incident_solutions_for_file()] - END - completed in '17.805967807769775s:  - App: 'coolstore', File: 'src/main/java/com/redhat/coolstore/service/InventoryNotificationMDB.java' with 2 incidents'

The difference of this run versus my first is this run is from make run-server using my python venv
The first run was using podman compose up

[jwmatthews]

I think this behavior is only seen with podman compose up because that flow will:

• Build in the kai/config.toml into the image

  • The default setting in [kai/config.toml[(https://github.com/konveyor/kai/blob/main/kai/config.toml#L27) is

    [models]
    provider = "ChatIBMGenAI"
    
    [models.args]
    model_id = "meta-llama/llama-3-70b-instruct"
    parameters.max_new_tokens = 2048
    

• Above setting has: model_id = "meta-llama/llama-3-70b-instruct"

• Next, I run with the presence of a build/config.toml

    [models]
    provider = "ChatOpenAI"
    
    [models.args]
    model = "gpt-4o"
  

• The build/config.toml I run with is passed in as a command line argument via the entrypoint.sh

• I suspect our behavior is to meld | merge these 2 files.... so we:

  1. use the config values from build/config.toml has highest
  2. use values from environment variables
  3. use values from inside the containers kai/config.toml

• With the above we end up with the config values for the model of

  models=KaiConfigModels(provider='ChatOpenAI', args={'model_id': 'meta-llama/llama-3-70b-instruct', 'parameters': {'max_new_tokens': 2048}, 'model': 'gpt-4o'},
  

[JonahSussman]

Pydantic calls a deep update method on each of its sources instead of overriding them. I see a couple of paths forward:

1. Remove all defaults for model config, and make it so the model is always required to be set by the user. That way, it's the last in the chain of precedence.
2. Create a custom pydantic source that changes the default behavior from deep update to overriding.