question: Is it possible to load Models from local drive?

[Mumpitz]

I am looking into building an app, that should run even without internet connection in isolated environments.
Therefore i need to load models locally.

I could not exactly see if this will be possible through the providers.
If not, do you plan on adding this?

[ThierryBleau]

You can't use local models yet but its definitely a planned feature!

We'll probably start by integrating Burn and Candle.

[cvauclair]

Hey @Mumpitz, if you already have a local LLM implemented in Rust, you can wrap that model in a struct which implements the CompletionModel trait. You could then create agents using that struct!

E.g.:

struct MyLocalModel {
    // Your local model
}

impl rig::completion::CompletionModel for MyLocalModel {
    // TODO
}

// Create an instance of your local model
let local_model = MyLocalModel::new(...);

// Create an agent that uses your custom local model
let agent = AgentBuilder::new(local_model)
    .preamble(...)
    .tools(...)
    .build();

let response = agent.prompt(...).await?;

However as @ThierryBleau said, Rig does not have an out-of-the-box solution that supports this at the moment. Hope this helps!

[awdemos]

Bumping this thread in the hope we can see some traction on a Llama provider.

[mateobelanger]

Will be adressed in a future version. This is one of our short-term priority.
Closing this to continue the discussion in #143.