[EPIC] Model support dashboard (v2)

[mudler]

This epic is a major tracker for all the backends additions that should be part of LocalAI v2 and ongoing efforts.

The objective is to release a v2 which deprecates old models which are now superseded, plus adding a new set. In order to achieve this my idea is to clean up the current state and start pinning dependencies for all the backends which requires specific environment settings (python-based ones).

Some key points:

• The go-llama backend will be sunset by the c++ version which is directly tied and easier to pinpoint to llama.cpp versions
• we will deprecate some (if not all) the duplicated backends arch (like falcon,gpt-2, gpt-neo-x, etc..) which are now supported by llama.cpp
• the python backends will use conda envionments to pin specific versions
• having a base point for backends for several languages (rust, c++, python, golang, ...)
• re-order internal layout to accomodate more backends as needed

Some backends will be deprecated as superseded, as such some repositories will be archived (TBD yet).

Backends:

• feat: add support for GPTQ-for-LLaMa #1055
• feat: Add support for LLaVA (text+images) and miniGPT4 #1054
• feat(exllama2): Add support to exllama2 #1053
• feat: tortoiseTTS #1016
• feat(transformer): transformer-based backend #1015
• [backend] llama-cpp, C++ gRPC backend #1154
• feat(alias): alias llama to llama-cpp, update docs #1448
• img2vid #1348
• feat(Petals): Add support to Petals #1262
• feat: split core and extra images with additional backends #1162
• feat(vllm): Initial vllm backend implementation #1026
• vall-e-x tts fails with ModuleNotFoundError: No module named 'utils' #1174
• typo in vall-e-x backend. #1370
• SDXL turbo #1380
• exllama(v2): fix exllamav1, add exllamav2 #1384
• feat(img2vid,txt2vid): Initial support for img2vid,txt2vid #1442
• coqui #1464
• feat: add transformers-musicgen backend #1387
• s4 mamba support #1588
• Localai v2 Image2Image not working #1392
• Support for Microsoft Phi-2 model #1438
• Vcoder #1529
• https://github.com/myshell-ai/OpenVoice #1639
• Support DeepSpeed FastGen #1538
• https://github.com/VikParuchuri/surya
• Backend: MLX #1638
• Model: whisperspeech #1636
• https://github.com/magic-research/magic-animate
• Support: moondream #1637
• feat(outlines): Add support pf outlines #1261
• Backend (lavie): Text to video #1640
• feat(deepSpare): Add support of DeepSpare #1263
• SAM support #915
• feature: support CTranslate2 #839
• feat: add support to ctransformers #1056
• feat: support chatglm #754
• feature: rust https://github.com/rustformers backend #939
• Support Qwen Model by Alibaba #1110
• Ollama support #1028
• https://github.com/chenxwh/insanely-fast-whisper
• https://twitter.com/ggerganov/status/1724898446770069983
• https://llm.mlc.ai/docs/deploy/python.html
• vit.cpp
• https://github.com/penghao-wu/vstar
• model pre-caching (huggingface/transformers) #1497
• model hotload (prepare it in ram before API is ready) #1539
• Tinygrad ( https://github.com/tinygrad/tinygrad/blob/master/examples/llama3.py )
• diffusers: add esrgan #917
• Diarization endpoint #1648
• Tokenization endpoint #1649
• backend(rwkv): python rwkv backend #1658
• Intel GPU support #1653
• feat(llama.cpp): Vulkan, Kompute, SYCL #1647
• Better Support for AMD and ROCM via docker containers. #1592
• https://github.com/apple/ml-mgie
• https://github.com/Zyphra/BlackMamba
• https://github.com/PKU-YuanGroup/MoE-LLaVA
• https://github.com/PKU-YuanGroup/Video-LLaVA
• https://github.com/LargeWorldModel/LWM
• https://github.com/Stability-AI/StableCascade/blob/master/inference%2Ftext_to_image.ipynb
• feat. add OpenVINO Model Server as a Backend #1722
• https://github.com/myshell-ai/MeloTTS
• [feature] TTS support request #1819
• https://github.com/roboflow/supervision
• https://github.com/Rudrabha/Wav2Lip
• https://github.com/jasonppy/VoiceCraft
• https://github.com/PKU-YuanGroup/Open-Sora-Plan
• feat: llamafile backend #2010
• Please support reranker API #2018
• feat(parler-tts): Add new backend #2027
• https://huggingface.co/nateraw/musicgen-songstarter-v0.2?s=09
• https://github.com/PABannier/bark.cpp
• feat(multimodal): Video understanding #2318
• https://github.com/systran/faster-whisper
• integrate whisperX #3375
• Please integrate chatTTS #2441
• https://github.com/fishaudio/fish-speech
• https://github.com/ictnlp/LLaMA-Omni
• Add support for cogvideo models #3628
• Moshi https://github.com/kyutai-labs/moshi/blob/main/moshi%2FREADME.md
• feat(multimodal): Audio understanding #3602
• https://github.com/baaivision/Emu3
• https://github.com/lucidrains/e2-tts-pytorch
• https://github.com/SWivid/F5-TTS
• https://github.com/homebrewltd/ichigo
• https://github.com/jy0205/Pyramid-Flow
• Support for SpiritLM #3966

After conda

We need to still test on master:

• Vall-E-X
• diffusers
• bark
• autogptq
• exllama
• vllm

Some rough first-steps required :

• docs: add documentation on how to write LocalAI backends #1057
• [backend] llama-cpp, C++ gRPC backend #1154
• feat(conda): conda environments #1144 (conda environments) (enhancement: use separate conda env for extra grpc backends #1117)
• ci: GPU tests #1116 (GPU testing)
• llama-cpp: feat parity with go-llama #1255 ( Pending items from [backend] llama-cpp, C++ gRPC backend #1154 )

Re-layout

• refactor: move backends into the backends directory #1279
• feat: move extra inside backends #1264

After the re-layout we can add new backends listed above without any clashes and:

• sunset the current llama in favor of llama-cpp
  • llama-cpp: feat parity with go-llama #1255
• refactor: rename llama-stable to llama-ggml #1287

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

Issue description:
The issue is related to the LocalAI project and its implementation of the Image endpoint. The user wants to create a cute baby sea otter image with the given prompt, size, and number of images. However, the code provided in the readInput function is not able to parse the input correctly and returns an error.

Proposed solution:
The issue seems to be related to the parsing of the input parameters. The user can try modifying the code in the readInput function to handle the specific format of the prompt, size, and number of images provided by the OpenAI API. Alternatively, they can try using a different API that provides more flexibility in the input format or use a different tool to generate the image.

If the issue persists, the user can also check if there are any updates available for the LocalAI project or report the issue on the GitHub repository to get help from the community.

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/CONTRIBUTING.md
• https://localai.io/index.html
• https://github.com/go-skynet/LocalAI/tree/master/go.mod
• https://github.com/go-skynet/LocalAI/tree/master/api/openai/image.go

[Aisuko]

I am thinking here, Should we need to have a outline likes below:

• C++
  • llama.cpp series LLM, like: Qwen, chatglm)
• Python
  • [ ]Huggingface
  • autoGPT
• Rust
  • rustformers
  • candle

If we keep this in mind, we can split the model to the specific backend. For example, I saw the Qwen LLM mentioned their C++ implementation of it "working in the same way as llama.cpp". So, maybe it can be loaded by using llama.cpp. It should be compatible with C++ our backend.

I also suggest to add some labels are related to the series of the backends. We can know a new model can be compatible by our backends.

[mudler]

conda branch was merged in #1144 . I'm looking now into make the llama.cpp backend on par with llama-go and also add llava support to it.

I'm going to refactor and re-layout things in the new backend directory too

[Aisuko]

@mudler thank you for mentioning. Here are some questions(second, and third one) may need you help #1180 (comment). Here what I am thinking is that we can use a tiny model to test the Rust backend features. And make sure everything ok. Maybe we can merge it.

And if everything ok. We can add other LLMs. I have plan to support Llama2(60% finished but it still has an issue), whisper, and also support onnex format.

[mudler]

Breaking re-layout PR: #1279

[kno10]

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

[muka]

I may start looking into #1273 while this progresses. What do you think ?

[mudler]

I may start looking into #1273 while this progresses. What do you think ?

please feel free to go ahead, there are many pieces involved in here, any help is more than appreciated 👍

[mudler]

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

in #1746 I'm taking care of automatically binding the HF cache variables to the models directories if not set already