Skip to content

Commit

Permalink
Apply suggestions from code review
Browse files Browse the repository at this point in the history
Signed-off-by: redoomed1 <[email protected]>
  • Loading branch information
redoomed1 authored Nov 13, 2024
1 parent ea9444c commit c69191b
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 12 deletions.
20 changes: 9 additions & 11 deletions docs/ai-chat.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@ To run AI locally, you need both an AI model and an AI client.

### Find and Choose a Model

There are many permissively licensed **models available to download**. **[Hugging Face](https://huggingface.co/models?library=gguf)** is a platform that lets you browse, research, and download models in common formats like GGUF. Companies that provide good open-weights models include big names like Mistral, Meta, Microsoft, and Google. But there are also many community models and 'fine-tunes' available. For consumer-grade hardware, it is generally recommended to use [quantized models](https://huggingface.co/docs/optimum/en/concept_guides/quantization) for the best balance between model quality and performance.
There are many permissively licensed models available to download. **[Hugging Face](https://huggingface.co/models?library=gguf)** is a platform that lets you browse, research, and download models in common formats like GGUF. Companies that provide good open-weights models include big names like Mistral, Meta, Microsoft, and Google. However, there are also many community models and 'fine-tunes' available. As mentioned above, [quantized models](https://huggingface.co/docs/optimum/en/concept_guides/quantization) offer the best balance between model quality and performance for those using consumer-grade hardware.

To help you choose a model that fits your needs, you can look at leaderboards and benchmarks. The most widely-used leaderboard is [LM Arena](https://lmarena.ai/), a "Community-driven Evaluation for Best AI chatbots". There is also the [OpenLLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard), which focus on the performance of open-weights models on common benchmarks like MMLU-PRO. However, there are also specialized benchmarks which measure factors like [emotional intelligence](https://eqbench.com/), ["uncensored general intelligence"](https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard), and many [others](https://www.nebuly.com/blog/llm-leaderboards).

### Model Security

When you have found an AI model of your liking, you should download it in a safe manner. When you use an AI client that maintains their own library of model files (such as [Ollama](#ollama) and [Llamafile](#llamafile)), you should download it from there. However, if you want to download models not present in their library, or use an AI client that doesn't maintain its library (such as [Kobold.cpp](#koboldcpp)), you will need to take extra steps to ensure that the AI model you download is safe and legitimate.
When you have found an AI model of your liking, you should download it in a safe manner. When you use an AI client that maintains their own library of model files (such as [Ollama](#ollama-cli) and [Llamafile](#llamafile)), you should download it from there. However, if you want to download models not present in their library, or use an AI client that doesn't maintain its library (such as [Kobold.cpp](#koboldcpp)), you will need to take extra steps to ensure that the AI model you download is safe and legitimate.

We recommend downloading model files from Hugging Face, as it provides several features to verify that your download is genuine and safe to use.

Expand All @@ -69,7 +69,7 @@ A downloaded model is generally safe if they satisfy all of the above checks.
| Local Client | GPU Support | Image Generation | Speech Recognition | Automatically Downloaded Models | Custom Parameters |
|---|---|---|---|---|---|
| [Kobold.cpp](#koboldcpp) | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-check:{ .pg-green } |
| [Ollama](#ollama) | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | :material-check:{ .pg-green } | :material-close:{ .pg-red } |
| [Ollama](#ollama-cli) | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | :material-check:{ .pg-green } | :material-close:{ .pg-red } |
| [Llamafile](#llamafile) | :material-check:{ .pg-green } | :material-close:{ .pg-red } | :material-close:{ .pg-red } | :material-alert-outline:{ .pg-orange } Few models available | :material-alert-outline:{ .pg-orange } |

### Kobold.cpp
Expand Down Expand Up @@ -107,19 +107,19 @@ Kobold.cpp might not run on computers without AVX/AVX2 support.

</div>

### Ollama
### Ollama (CLI)

<div class="admonition recommendation" markdown>

![Ollama Logo](assets/img/ai-chat/ollama.svg){align=right}

Ollama is an easy-to-use command-line AI assistant that is available on macOS, Linux, and Windows.
Ollama is a command-line AI assistant that is available on macOS, Linux, and Windows. Ollama is a great choice if you're looking for an AI client that's easy-to-use and widely compatible. It also doesn't involve any manual setup, while still using inference and other techniques to make outputs faster.

In addition to supporting a wide range of text models, Ollama also supports [LLaVA](https://github.com/haotian-liu/LLaVA) models and has *experimental* support for Meta's [Llama vision capabilities](https://huggingface.co/blog/llama32#what-is-llama-32-vision).
In addition to supporting a wide range of text models, Ollama also supports [LLaVA](https://github.com/haotian-liu/LLaVA) models and has experimental support for Meta's [Llama vision capabilities](https://huggingface.co/blog/llama32#what-is-llama-32-vision).

[:octicons-home-16: Homepage](https://github.com/ollama/ollama){ .md-button .md-button--primary }
[:octicons-info-16:](https://github.com/ollama/ollama){ .card-link title="Documentation" }
[:octicons-code-16:](https://github.com/ollama/ollama){ .card-link title="Source Code"}
[:octicons-info-16:](https://github.com/ollama/ollama#readme){ .card-link title="Documentation" }
[:octicons-code-16:](https://github.com/ollama/ollama){ .card-link title="Source Code" }

<details class="downloads" markdown>
<summary>Downloads</summary>
Expand All @@ -132,9 +132,7 @@ In addition to supporting a wide range of text models, Ollama also supports [LLa

</div>

Ollama is best if you're looking for an AI client that has great compatibility and ease of use. It runs on all desktop platforms and doesn't involve any manual setup, while still using inference and other techniques to make outputs faster.

It also simplifies the process of setting up a local AI chat, as it downloads the AI model you want to use automatically. For example, running `ollama run llama3.2` will automatically download and run the Llama 3.2 model. Furthermore, ollama maintains their own [model library](https://ollama.com/library/) where they host the files of various AI models. This ensures models are vetted for both performance and security, eliminating the need to manually verify model authenticity.
Ollama simplifies the process of setting up a local AI chat, as it downloads the AI model you want to use automatically. For example, running `ollama run llama3.2` will automatically download and run the Llama 3.2 model. Furthermore, Ollama maintains their own [model library](https://ollama.com/library) where they host the files of various AI models. This ensures models are vetted for both performance and security, eliminating the need to manually verify model authenticity.

### Llamafile

Expand Down
2 changes: 1 addition & 1 deletion docs/tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -476,7 +476,7 @@ For encrypting your OS drive, we typically recommend using the encryption tool y

- ![Kobold logo](assets/img/ai-chat/kobold.png){ .twemoji loading=lazy }[Kobold.cpp](ai-chat.md#koboldcpp)
- ![Llamafile logo](assets/img/ai-chat/llamafile.svg){ .twemoji loading=lazy }[Llamafile](ai-chat.md#llamafile)
- ![Ollama logo](assets/img/ai-chat/ollama.svg){ .twemoji loading=lazy }[Ollama](ai-chat.md#ollama)
- ![Ollama logo](assets/img/ai-chat/ollama.svg){ .twemoji loading=lazy }[Ollama](ai-chat.md#ollama-cli)

</div>

Expand Down

0 comments on commit c69191b

Please sign in to comment.