Ollama / dspy.OllamaLocal support

[k-nearest-neighbor]

https://ollama.com/ is a way to run llama locally. It serves models at a local url.

I'd like to use it in dspy_nodes but maybe some modifications would be needed
i'd be happy to make a pull request if you have an idea of how it could be done and talk me through it a little.

There's an object for it in DSPy: https://dspy-docs.vercel.app/api/local_language_model_clients/Ollama

[tom-doerr]

The model node I currently use is pretty simple. If Ollama has an OpenAI compatible API, no changes to the code might be necessary. Maybe api_base should become a parameter.

from custom_nodes.dspy_nodes.nodes.global_file import server_settings
import dspy

class Model:     

    @classmethod
    def INPUT_TYPES(cls):
               
        return {"required": {       
                    "model": ("STRING", {"default": "microsoft/Phi-3-medium-128k-instruct"}),
                    "max_tokens": ("INT", {"default": 100, "min": 1}),
                    }
                }

    RETURN_TYPES = ("MODEL",)
    FUNCTION = "set_params"
    OUTPUT_NODE = True
    CATEGORY = "DSPy"

    def set_params(self, model, max_tokens):
        server_settings['model'] = model
        print("====== model file server_settings:", server_settings)
        # lm = dspy.HFClientVLLM(model=server_settings['model'], port=38242, url="http://localhost", max_tokens=200)
        # lm = dspy.HFClientVLLM(model=model, port=38242, url="http://localhost", max_tokens=200)
        lm = dspy.OpenAI(model=model, api_base="http://localhost:38242/v1/", api_key="EMPTY", max_tokens=max_tokens)
        return [lm]