Transition from Ollama to Hugging Face 🤗

[latekvo]

🤗

[latekvo]

Seems to be working, but requires further testing, and we have to decide what to do with the ollama setup.
I bet there must be a better way than just having a separate compose and containers all together.
We could remove ollama support completely IF - and this is a blocking requirement - if we recreate it's auto-allocation algorithm.
Currently llama takes a fixed amount of memory per config, and uses it to split the provided model and launch it accordingly.
We'd have to somehow calculate how many layers + context we can load at once, and apply it to the llama.cpp loader.
Then, we can confidently remove the inferior bridged solution ;)

[latekvo]

This modification turned out to require way to much additional work on the user's side, and upkeep of 2 separate systems from our side, as relying solely on bare bones llama.cpp is not as consistant as ollama.

I'm closing for now, but it's still up for consideration in future releases.