Add support for local models (OpenAI compatible) #157
Conversation
| message = [ | ||
| {"role": "system", "content": system_prompt}, | ||
| {"role": "user", "content": user_prompt} | ||
| ] |
There was a problem hiding this comment.
This may be a Jan-specific issue, but requests in format: {"type": "text", "text": system_prompt} do not work, since only text is supported (images are not). However, I left the image option toggle and the request format for that alone, in case any OpenAI compatible servers do support images.
| base_url = None | ||
|
|
||
| if 'Local OpenAI' in translator_key: | ||
| base_url = self.settings.ui.llm_widgets['local_oai_url_input'].text() | ||
|
|
||
| if not base_url and translator_key == "Local OpenAI Server": | ||
| raise ValueError(f"Base URL not found for translator: {translator_key}") | ||
|
|
||
| return base_url |
There was a problem hiding this comment.
Added logic to pull the base_url in translator.py and pass it to the get_llm_client() helper, rather than have the helper need to access the settings pane.
|
|
||
| local_oai_model_input = MLineEdit() | ||
| local_oai_model_input.setFixedWidth(400) | ||
| local_oai_model_input.setPlaceholderText("llama3.1-8b-instruct") |
There was a problem hiding this comment.
Using llama3.1-8b-instruct as the default model.
There was a problem hiding this comment.
It seems there is a bug in the functionality for setting up the default local LLM model. I used the ollama as the backend with ollama serve command, pasted the URL of the ollama local API server, and left the Model ID field empty. However, when I tried to perform a translation, I encountered an error like "Model not found for translator: xxx". It appears that the default value is not being set correctly when retrieving the model name in translator.py at line 59.
Using any of the current LLMs for translation is expensive, since you pay per-token for API calls. This PR adds an option to allow for a "Local OpenAI server" to be used instead.
This should allow for using locally hosted LLMs using tools such as vLLM or Jan. I have had success translating pages using Llama-3.1-8B-Instruct through Jan. Importantly, I have not tried vLLM, as I am running an M1 Mac, but it should work. Anyone else able to verify this would be much appreciated.
Addresses issue #150
This PR should be functional, but is marked as draft as the README still needs to be updated.