Alternative Inference Engines (LlamaFile, PowerInfer)

[BradKML]

Not sure if those are just hype or not, but LLamaFile recently noted that they found a way to speed up CPU inference. Would be nice to do fast Chatbots with docs on laptop.

[debanjum]

Given LlamaFile exposes an OpenAI API compatible proxy. You can have Khoj use any Llamafile based chat model. See the Khoj OpenAI Proxy Docs for how to set it up

P.S: Not sure how I missed replying before, my notification for discussion topics maybe wonky

[BradKML]

Thanks for the help and the self-host support