Add llmaz as another platform to run llama.cpp on Kubernetes

[kerthcet]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Hi, llmaz is a platform to serve large language models on Kubernetes, llama.cpp is an vital part of it for CPU inference. Here's an example:

apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: qwen2-0-5b-gguf
spec:
  familyName: qwen2
  source:
    modelHub:
      modelID: Qwen/Qwen2-0.5B-Instruct-GGUF
      filename: qwen2-0_5b-instruct-q5_k_m.gguf

apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: qwen2-0-5b
spec:
  replicas: 1
  modelClaim:
    modelName: qwen2-0-5b-gguf
  backendConfig:
    name: llamacpp
    args:
    - -fa # use flash attention

This is all your need to do, then you can serve models with llama.cpp in Kubernetes, we also support multi-host inference, we'll try to integrate with llama.cpp in the near future. If you're interested, here's an example https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/llamacpp provided by the lws community.

Thanks!

[kerthcet]

kindly ping @ggerganov