Feature Request: Support for free and local LLMs (ollama)

[kieran-mace]

Ollama is a fantastic tool, that enables user to run freely available LLMs locally and chat with them via the command line. They regularly update what LLMs are available (llama3 became available this week)

My feature request is to enable the chattr app to interact with these local and open source instances.

[edgararuiz]

Hi! That's actually in the works, I have a branch that already has a working prototype. I just need to document it. Feel free to try it and provide feedback 😄

pak::pak("mlverse/chattr@ollama")
library(chattr)
chattr_use("ollama")
chattr("hi")

[ManuelSpinola]

Hi!

I followed your instructions but I cannot run Ollama.

pak::pak("mlverse/chattr@ollama")

ℹ No downloads are needed
✔ 1 pkg + 44 deps: kept 45 [1.1s]

chattr_use("ollama")

── chattr
• Provider: Ollama
• Path/URL: http://localhost:11434/
• Model: llama2
• Label: Ollama

chattr("hi")
Error in chattr():
! ! The 'llama2' model is not found.
Would you like to download it?
Backtrace:

1. chattr::chattr("hi")

[ga-it]

Rather than work specifically with ollama, could you allow defining the endpoint for the OpenAI connection?

In addition to the endpoint, this should also then allow selection of the specific model to connect to and the litellm api key.

I believe the OpenAI package does most of this. Inheriting this as the backend could allow chattr to focus on the front end RStudio / RShiny integration.

This would allow connection to LiteLLM and from there proxy connection to ollama or a variety of other LLMs

[ManuelSpinola]

Thank you. How I do that? *Manuel Spínola, Ph.D.* Instituto Internacional en Conservación y Manejo de Vida Silvestre Universidad Nacional Apartado 1350-3000 Heredia COSTA RICA ***@***.*** ***@***.***> ***@***.*** Teléfono: (506) 8706 - 4662 Sitio web institucional: ICOMVIS <http://www.icomvis.una.ac.cr/index.php/manuel> Sitio web personal: Sitio personal <https://mspinola-sitioweb.netlify.app> Blog sobre Ciencia de Datos: Blog de Ciencia de Datos <https://mspinola-ciencia-de-datos.netlify.app> El El mar, 18 jun 2024 a la(s) 07:11, ga-it ***@***.***> escribió:

…

Rather than work specifically with ollama, could you allow defining the endpoint for the OpenAI connection? This would allow connection to LiteLLM and from there proxy connection to ollama or a variety of other LLMs — Reply to this email directly, view it on GitHub <#95 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI3FB535XV5VQO3VJ4X6WLZIAWXNAVCNFSM6AAAAABHC5D5LGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZWGA3DQMJUGY> . You are receiving this because you commented.Message ID: ***@***.***>

[ManuelSpinola]

Now, if I follow: ***@***.***") library(chattr) chattr_use("ollama") chattr("hi") I can work with ollama El mar, 18 jun 2024 a las 7:11, ga-it ***@***.***>) escribió:

…

Rather than work specifically with ollama, could you allow defining the endpoint for the OpenAI connection? This would allow connection to LiteLLM and from there proxy connection to ollama or a variety of other LLMs — Reply to this email directly, view it on GitHub <#95 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI3FB535XV5VQO3VJ4X6WLZIAWXNAVCNFSM6AAAAABHC5D5LGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZWGA3DQMJUGY> . You are receiving this because you commented.Message ID: ***@***.***>

-- *Manuel Spínola, Ph.D.* Instituto Internacional en Conservación y Manejo de Vida Silvestre Universidad Nacional Apartado 1350-3000 Heredia COSTA RICA ***@***.*** ***@***.***> ***@***.*** Teléfono: (506) 8706 - 4662 Sitio web institucional: ICOMVIS <http://www.icomvis.una.ac.cr/index.php/manuel> Sitio web personal: Sitio personal <https://mspinola-sitioweb.netlify.app> Blog sobre Ciencia de Datos: Blog de Ciencia de Datos <https://mspinola-ciencia-de-datos.netlify.app>

[ga-it]

Apologies - to clarify I was suggesting an alternative approach to @mlverse for the enhancement

[ManuelSpinola]

No problem! *Manuel Spínola, Ph.D.* Instituto Internacional en Conservación y Manejo de Vida Silvestre Universidad Nacional Apartado 1350-3000 Heredia COSTA RICA ***@***.*** ***@***.***> ***@***.*** Teléfono: (506) 8706 - 4662 Sitio web institucional: ICOMVIS <http://www.icomvis.una.ac.cr/index.php/manuel> Sitio web personal: Sitio personal <https://mspinola-sitioweb.netlify.app> Blog sobre Ciencia de Datos: Blog de Ciencia de Datos <https://mspinola-ciencia-de-datos.netlify.app> El El mar, 18 jun 2024 a la(s) 08:12, ga-it ***@***.***> escribió:

…

Apologies - to clarify I was suggesting an alternative approach to @mlverse <https://github.com/mlverse> for the enhancement — Reply to this email directly, view it on GitHub <#95 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI3FB3XSAHVWSVWHH3QKLTZIA56PAVCNFSM6AAAAABHC5D5LGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZWGIYDSMBSGU> . You are receiving this because you commented.Message ID: ***@***.***>

[alexvorobiev]

@edgararuiz Is there a roadmap to merge the branch? Thanks!

[jansoe]

Rather than work specifically with ollama, could you allow defining the endpoint for the OpenAI connection?

I would really appreciate this. Our university runs an open-ai API compatible server (vLLM). It would be great to use it by just setting the model endpoint url

[wangqc0]

Hi! That's actually in the works, I have a branch that already has a working prototype. I just need to document it. Feel free to try it and provide feedback 😄

pak::pak("mlverse/chattr@ollama")
library(chattr)
chattr_use("ollama")
chattr("hi")

Thank you very much for sharing this! I have deployed a local model on my laptop using Ollama. I am trying to revise your "ollama.yml" file (change the model to the model I want) and build the package on my laptop. However, when I try to use it on R, it shows:

! The '<THE MODEL I USE>' model is not found.
Would you like to download it?

1: Yes
2: No

Every time I choose "1", and it pulls (modify my local model file without changing anything), then I can use it once. From the speed, I can tell that the system does not download anything. If I want to use it again (either with chattr_app() via Viewer box or chattr("<MY QUESTION>")), the "not found" message reappears, and I have to repeat the above steps again. Could you fix the issue?

[liar666]

Thanks for the nice work.

• Same problem as above: I need to ask to download the model each time :(
• It would be nice to be able to stop the generation, e.g. have everything run in a separate thread,
  (I tried to use a 1.5b local version of DeepSeek R1 that started "thinking" at length, which blocked my work for a loooooong time :( )
• Would be nice to be able to change the model from the 'setup' GUI (based on ollama list output) rather than finding & editing a yaml file.