From 6903076ce4e3d3060c056403bb4bf09ac5115e39 Mon Sep 17 00:00:00 2001
From: gitlawr <lawrleegle@gmail.com>
Date: Tue, 17 Dec 2024 19:07:03 +0800
Subject: [PATCH] Deployed 0b1dbbb to 0.4 with MkDocs 1.6.0 and mike 2.1.3

---
 0.4/development/index.html                    |   2 +-
 .../installation-requirements/index.html      |   2 +-
 .../manual-installation/index.html            |   2 +-
 0.4/search/search_index.json                  |   2 +-
 0.4/sitemap.xml                               |  94 +++++++++---------
 0.4/sitemap.xml.gz                            | Bin 712 -> 712 bytes
 .../index.html                                |   2 +-
 7 files changed, 52 insertions(+), 52 deletions(-)

diff --git a/0.4/development/index.html b/0.4/development/index.html
index 119d345..6706655 100644
--- a/0.4/development/index.html
+++ b/0.4/development/index.html
@@ -1564,7 +1564,7 @@
 
 <h1 id="development-guide">Development Guide</h1>
 <h2 id="prerequisites">Prerequisites</h2>
-<p>Install <code>python 3.10+</code>.</p>
+<p>Install Python (version 3.10 to 3.12).</p>
 <h2 id="set-up-environment">Set Up Environment</h2>
 <div class="highlight"><pre><span></span><code>make<span class="w"> </span>install
 </code></pre></div>
diff --git a/0.4/installation/installation-requirements/index.html b/0.4/installation/installation-requirements/index.html
index bce2148..12427dd 100644
--- a/0.4/installation/installation-requirements/index.html
+++ b/0.4/installation/installation-requirements/index.html
@@ -1773,7 +1773,7 @@
 <h1 id="installation-requirements">Installation Requirements</h1>
 <p>This page describes the software and networking requirements for the nodes where GPUStack will be installed.</p>
 <h2 id="python-requirements">Python Requirements</h2>
-<p>GPUStack requires Python version 3.10 or higher.</p>
+<p>GPUStack requires Python version 3.10 to 3.12.</p>
 <h2 id="operating-system-requirements">Operating System Requirements</h2>
 <p>GPUStack is supported on the following operating systems:</p>
 <ul class="task-list">
diff --git a/0.4/installation/manual-installation/index.html b/0.4/installation/manual-installation/index.html
index 23a47c1..77bab86 100644
--- a/0.4/installation/manual-installation/index.html
+++ b/0.4/installation/manual-installation/index.html
@@ -1640,7 +1640,7 @@
 
 <h1 id="manual-installation">Manual Installation</h1>
 <h2 id="prerequites">Prerequites:</h2>
-<p>Install python3.10 or above with pip.</p>
+<p>Install Python version 3.10 to 3.12.</p>
 <h2 id="install-gpustack-cli">Install GPUStack CLI</h2>
 <p>Run the following to install GPUStack:</p>
 <div class="highlight"><pre><span></span><code><span class="c1"># You can add extra dependencies, options are &quot;vllm&quot;, &quot;audio&quot; and &quot;all&quot;.</span>
diff --git a/0.4/search/search_index.json b/0.4/search/search_index.json
index f364e91..97a9dbf 100644
--- a/0.4/search/search_index.json
+++ b/0.4/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"api-reference/","title":"API Reference","text":"<p>GPUStack provides a built-in Swagger UI. You can access it by navigating to <code>&lt;gpustack-server-url&gt;/docs</code> in your browser to view and interact with the APIs.</p> <p></p>"},{"location":"architecture/","title":"Architecture","text":"<p>The following diagram shows the architecture of GPUStack:</p> <p></p>"},{"location":"architecture/#server","title":"Server","text":"<p>The GPUStack server consists of the following components:</p> <ul> <li>API Server: Provides a RESTful interface for clients to interact with the system. It handles authentication and authorization.</li> <li>Scheduler: Responsible for assigning model instances to workers.</li> <li>Model Controller: Manages the rollout and scaling of model instances to match the desired model replicas.</li> <li>HTTP Proxy: Routes completion API requests to backend inference servers.</li> </ul>"},{"location":"architecture/#worker","title":"Worker","text":"<p>GPUStack workers are responsible for:</p> <ul> <li>Running inference servers for model instances assigned to the worker.</li> <li>Reporting status to the server.</li> </ul>"},{"location":"architecture/#sql-database","title":"SQL Database","text":"<p>The GPUStack server connects to a SQL database as the datastore. GPUStack uses SQLite by default, but you can configure it to use an external PostgreSQL as well.</p>"},{"location":"architecture/#inference-server","title":"Inference Server","text":"<p>Inference servers are the backends that performs the inference tasks. GPUStack supports llama-box, vLLM and vox-box as the inference server.</p>"},{"location":"architecture/#rpc-server","title":"RPC Server","text":"<p>The RPC server enables running llama-box backend on a remote host. The Inference Server communicates with one or several instances of RPC server, offloading computations to these remote hosts. This setup allows for distributed LLM inference across multiple workers, enabling the system to load larger models even when individual resources are limited.</p>"},{"location":"code-of-conduct/","title":"Contributor Code of Conduct","text":""},{"location":"code-of-conduct/#our-pledge","title":"Our Pledge","text":"<p>We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.</p> <p>We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.</p>"},{"location":"code-of-conduct/#our-standards","title":"Our Standards","text":"<p>Examples of behavior that contributes to a positive environment for our community include:</p> <ul> <li>Demonstrating empathy and kindness toward other people</li> <li>Being respectful of differing opinions, viewpoints, and experiences</li> <li>Giving and gracefully accepting constructive feedback</li> <li>Accepting responsibility and apologizing to those affected by our mistakes,   and learning from the experience</li> <li>Focusing on what is best not just for us as individuals, but for the overall   community</li> </ul> <p>Examples of unacceptable behavior include:</p> <ul> <li>The use of sexualized language or imagery, and sexual attention or advances of   any kind</li> <li>Trolling, insulting or derogatory comments, and personal or political attacks</li> <li>Public or private harassment</li> <li>Publishing others' private information, such as a physical or email address,   without their explicit permission</li> <li>Other conduct which could reasonably be considered inappropriate in a   professional setting</li> </ul>"},{"location":"code-of-conduct/#enforcement-responsibilities","title":"Enforcement Responsibilities","text":"<p>Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.</p> <p>Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.</p>"},{"location":"code-of-conduct/#scope","title":"Scope","text":"<p>This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.</p>"},{"location":"code-of-conduct/#enforcement","title":"Enforcement","text":"<p>Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at contact@gpustack.ai. All complaints will be reviewed and investigated promptly and fairly.</p> <p>All community leaders are obligated to respect the privacy and security of the reporter of any incident.</p>"},{"location":"code-of-conduct/#enforcement-guidelines","title":"Enforcement Guidelines","text":"<p>Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:</p>"},{"location":"code-of-conduct/#1-correction","title":"1. Correction","text":"<p>Community Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.</p> <p>Consequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.</p>"},{"location":"code-of-conduct/#2-warning","title":"2. Warning","text":"<p>Community Impact: A violation through a single incident or series of actions.</p> <p>Consequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.</p>"},{"location":"code-of-conduct/#3-temporary-ban","title":"3. Temporary Ban","text":"<p>Community Impact: A serious violation of community standards, including sustained inappropriate behavior.</p> <p>Consequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.</p>"},{"location":"code-of-conduct/#4-permanent-ban","title":"4. Permanent Ban","text":"<p>Community Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.</p> <p>Consequence: A permanent ban from any sort of public interaction within the community.</p>"},{"location":"code-of-conduct/#attribution","title":"Attribution","text":"<p>This Code of Conduct is adapted from the Contributor Covenant, version 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.</p>"},{"location":"contributing/","title":"Contributing to GPUStack","text":"<p>Thanks for taking the time to contribute to GPUStack!</p> <p>Please review and follow the Code of Conduct.</p>"},{"location":"contributing/#filing-issues","title":"Filing Issues","text":"<p>If you find any bugs or are having any trouble, please search the reported issue as someone may have experienced the same issue, or we are actively working on a solution.</p> <p>If you can't find anything related to your issue, contact us by filing an issue. To help us diagnose and resolve, please include as much information as possible, including:</p> <ul> <li>Software: GPUStack version, installation method, operating system info, etc.</li> <li>Hardware: Node info, GPU info, etc.</li> <li>Steps to reproduce: Provide as much detail on how you got into the reported situation.</li> <li>Logs: Please include any relevant logs, such as server logs, worker logs, etc.</li> </ul>"},{"location":"contributing/#contributing-code","title":"Contributing Code","text":"<p>For setting up development environment, please refer to Development Guide.</p> <p>If you're fixing a small issue, you can simply submit a PR. However, if you're planning to submit a bigger PR to implement a new feature or fix a relatively complex bug, please open an issue that explains the change and the motivation for it. If you're addressing a bug, please explain how to reproduce it.</p>"},{"location":"contributing/#updating-documentation","title":"Updating Documentation","text":"<p>If you have any updates to our documentation, feel free to file an issue with the <code>documentation</code> label or make a pull request.</p>"},{"location":"development/","title":"Development Guide","text":""},{"location":"development/#prerequisites","title":"Prerequisites","text":"<p>Install <code>python 3.10+</code>.</p>"},{"location":"development/#set-up-environment","title":"Set Up Environment","text":"<pre><code>make install\n</code></pre>"},{"location":"development/#run","title":"Run","text":"<pre><code>poetry run gpustack\n</code></pre>"},{"location":"development/#build","title":"Build","text":"<pre><code>make build\n</code></pre> <p>And check artifacts in <code>dist</code>.</p>"},{"location":"development/#test","title":"Test","text":"<pre><code>make test\n</code></pre>"},{"location":"development/#update-dependencies","title":"Update Dependencies","text":"<pre><code>poetry add &lt;something&gt;\n</code></pre> <p>Or</p> <pre><code>poetry add --group dev &lt;something&gt;\n</code></pre> <p>For dev/testing dependencies.</p>"},{"location":"overview/","title":"GPUStack","text":"<p>GPUStack is an open-source GPU cluster manager for running AI models.</p>"},{"location":"overview/#key-features","title":"Key Features","text":"<ul> <li>Broad Hardware Compatibility: Run with different brands of GPUs in Apple MacBooks, Windows PCs, and Linux servers.</li> <li>Broad Model Support: From LLMs to diffusion models, audio, embedding, and reranker models.</li> <li>Scales with Your GPU Inventory: Easily add more GPUs or nodes to scale up your operations.</li> <li>Distributed Inference: Supports both single-node multi-GPU and multi-node inference and serving.</li> <li>Multiple Inference Backends: Supports llama-box (llama.cpp &amp; stable-diffusion.cpp), vox-box and vLLM as the inference backend.</li> <li>Lightweight Python Package: Minimal dependencies and operational overhead.</li> <li>OpenAI-compatible APIs: Serve APIs that are compatible with OpenAI standards.</li> <li>User and API key management: Simplified management of users and API keys.</li> <li>GPU metrics monitoring: Monitor GPU performance and utilization in real-time.</li> <li>Token usage and rate metrics: Track token usage and manage rate limits effectively.</li> </ul>"},{"location":"overview/#supported-platforms","title":"Supported Platforms","text":"<ul> <li> macOS</li> <li> Windows</li> <li> Linux</li> </ul>"},{"location":"overview/#supported-accelerators","title":"Supported Accelerators","text":"<ul> <li> Apple Metal (M-series chips)</li> <li> NVIDIA CUDA (Compute Capability 6.0 and above)</li> <li> Ascend CANN</li> <li> Moore Threads MUSA</li> </ul> <p>We plan to support the following accelerators in future releases.</p> <ul> <li> AMD ROCm</li> <li> Intel oneAPI</li> <li> Qualcomm AI Engine</li> </ul>"},{"location":"overview/#supported-models","title":"Supported Models","text":"<p>GPUStack uses llama-box (bundled llama.cpp and stable-diffusion.cpp server), vLLM and vox-box as the backends and supports a wide range of models. Models from the following sources are supported:</p> <ol> <li> <p>Hugging Face</p> </li> <li> <p>ModelScope</p> </li> <li> <p>Ollama Library</p> </li> <li> <p>Local File Path</p> </li> </ol>"},{"location":"overview/#example-models","title":"Example Models:","text":"Category Models Large Language Models(LLMs) Qwen, LLaMA, Mistral, Deepseek, Phi, Yi Vision Language Models(VLMs) Llama3.2-Vision, Pixtral , Qwen2-VL, LLaVA, InternVL2 Diffusion Models Stable Diffusion, FLUX Rerankers GTE, BCE, BGE, Jina Audio Models Whisper (speech-to-text), CosyVoice (text-to-speech) <p>For full list of supported models, please refer to the supported models section in the inference backends documentation.</p>"},{"location":"overview/#openai-compatible-apis","title":"OpenAI-Compatible APIs","text":"<p>GPUStack serves OpenAI compatible APIs. For details, please refer to OpenAI Compatible APIs</p>"},{"location":"quickstart/","title":"Quickstart","text":""},{"location":"quickstart/#installation","title":"Installation","text":""},{"location":"quickstart/#linux-or-macos","title":"Linux or macOS","text":"<p>GPUStack provides a script to install it as a service on systemd or launchd based systems. To install GPUStack using this method, just run:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s -\n</code></pre>"},{"location":"quickstart/#windows","title":"Windows","text":"<p>Run PowerShell as administrator (avoid using PowerShell ISE), then run the following command to install GPUStack:</p> <pre><code>Invoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre>"},{"location":"quickstart/#other-installation-methods","title":"Other Installation Methods","text":"<p>For manual installation, docker installation or detailed configuration options, please refer to the Installation Documentation.</p>"},{"location":"quickstart/#getting-started","title":"Getting Started","text":"<ol> <li>Run and chat with the llama3.2 model:</li> </ol> <pre><code>gpustack chat llama3.2 \"tell me a joke.\"\n</code></pre> <ol> <li>Run and generate an image with the stable-diffusion-v3-5-large-turbo model:</li> </ol> <p>Tip</p> <p>This command downloads the model (~12GB) from Hugging Face. The download time depends on your network speed. Ensure you have enough disk space and VRAM (12GB) to run the model. If you encounter issues, you can skip this step and move to the next one.</p> <pre><code>gpustack draw hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF:stable-diffusion-v3-5-large-turbo-Q4_0.gguf \\\n\"A minion holding a sign that says 'GPUStack'. The background is filled with futuristic elements like neon lights, circuit boards, and holographic displays. The minion is wearing a tech-themed outfit, possibly with LED lights or digital patterns. The sign itself has a sleek, modern design with glowing edges. The overall atmosphere is high-tech and vibrant, with a mix of dark and neon colors.\" \\\n--sample-steps 5 --show\n</code></pre> <p>Once the command completes, the generated image will appear in the default viewer. You can experiment with the prompt and CLI options to customize the output.</p> <p></p> <ol> <li>Open <code>http://myserver</code> in the browser to access the GPUStack UI. Log in to GPUStack with username <code>admin</code> and the default password. You can run the following command to get the password for the default setup:</li> </ol> <p>Linux or macOS</p> <pre><code>cat /var/lib/gpustack/initial_admin_password\n</code></pre> <p>Windows</p> <pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\initial_admin_password\" -Raw\n</code></pre> <ol> <li>Click <code>Playground</code> in the navigation menu. Now you can chat with the LLM in the UI playground.</li> </ol> <p></p> <ol> <li> <p>Click <code>API Keys</code> in the navigation menu, then click the <code>New API Key</code> button.</p> </li> <li> <p>Fill in the <code>Name</code> and click the <code>Save</code> button.</p> </li> <li> <p>Copy the generated API key and save it somewhere safe. Please note that you can only see it once on creation.</p> </li> <li> <p>Now you can use the API key to access the OpenAI-compatible API. For example, use curl as the following:</p> </li> </ol> <pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -d '{\n    \"model\": \"llama3.2\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a helpful assistant.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n      }\n    ],\n    \"stream\": true\n  }'\n</code></pre>"},{"location":"quickstart/#cleanup","title":"Cleanup","text":"<p>After you complete using the deployed models, you can go to the <code>Models</code> page in the GPUStack UI and delete the models to free up resources.</p>"},{"location":"scheduler/","title":"Scheduler","text":""},{"location":"scheduler/#summary","title":"Summary","text":"<p>The scheduler's primary responsibility is to calculate the resources required by models instance and to evaluate and select the optimal workers/GPUs for model instances through a series of strategies. This ensures that model instances can run efficiently. This document provides a detailed overview of the policies and processes used by the scheduler.</p>"},{"location":"scheduler/#scheduling-process","title":"Scheduling Process","text":""},{"location":"scheduler/#filtering-phase","title":"Filtering Phase","text":"<p>The filtering phase aims to narrow down the available workers or GPUs to those that meet specific criteria. The main policies involved are:</p> <ul> <li>Label Matching Policy</li> <li>Status Policy</li> <li>Resource Fit Policy</li> </ul>"},{"location":"scheduler/#label-matching-policy","title":"Label Matching Policy","text":"<p>This policy filters workers based on the label selectors configured for the model. If no label selectors are defined for the model, all workers are considered. Otherwise, the system checks whether the labels of each worker node match the model's label selectors, retaining only those workers that match.</p>"},{"location":"scheduler/#status-policy","title":"Status Policy","text":"<p>This policy filters workers based on their status, retaining only those that are in a READY state.</p>"},{"location":"scheduler/#resource-fit-policy","title":"Resource Fit Policy","text":"<p>The Resource Fit Policy is a critical strategy in the scheduling system, used to filter workers or GPUs based on resource compatibility. The goal of this policy is to ensure that model instances can run on the selected nodes without exceeding resource limits. The Resource Fit Policy prioritizes candidates in the following order:</p> <ul> <li>Single Worker Node, Single GPU Full Offload: Identifies candidates where a single GPU on a single worker can fully offload the model, which usually offers the best performance.</li> <li>Single Worker Node, Multiple GPU Full Offload: Identifies candidates where multiple GPUs on a single worker can fully the offload the model.</li> <li>Single Worker Node Partial Offload: Identifies candidates on a single worker that can handle a partial offload, used only when partial offloading is allowed.</li> <li>Distributed Inference Across Multiple Workers: Identifies candidates where a combination of GPUs across multiple workers can handle full or partial offloading, used only when distributed inference across nodes is permitted.</li> <li>Single Worker Node, CPU: When no GPUs are available, the system will use the CPU for inference, identifying candidates where memory resources on a single worker are sufficient.</li> </ul>"},{"location":"scheduler/#scoring-phase","title":"Scoring Phase","text":"<p>The scoring phase evaluates the filtered candidates, scoring them to select the optimal deployment location. The primary strategy involved is:</p> <ul> <li>Placement Strategy Policy</li> </ul>"},{"location":"scheduler/#placement-strategy-policy","title":"Placement Strategy Policy","text":"<ul> <li>Binpack</li> </ul> <p>This strategy aims to \"pack\" as many model instances as possible into the fewest number of \"bins\" (e.g., Workers/GPUs) to optimize resource utilization. The goal is to minimize the number of bins used while maximizing resource efficiency, ensuring each bin is filled as efficiently as possible without exceeding its capacity. Model instances are placed in the bin with the least remaining space to minimize leftover capacity in each bin.</p> <ul> <li>Spread</li> </ul> <p>This strategy seeks to distribute multiple model instances across different worker nodes as evenly as possible, improving system fault tolerance and load balancing.</p>"},{"location":"troubleshooting/","title":"Troubleshooting","text":""},{"location":"troubleshooting/#view-gpustack-logs","title":"View GPUStack Logs","text":"<p>If you installed GPUStack using the installation script, you can view GPUStack logs at the following path:</p>"},{"location":"troubleshooting/#linux-or-macos","title":"Linux or macOS","text":"<pre><code>/var/log/gpustack.log\n</code></pre>"},{"location":"troubleshooting/#windows","title":"Windows","text":"<pre><code>\"$env:APPDATA\\gpustack\\log\\gpustack.log\"\n</code></pre>"},{"location":"troubleshooting/#configure-log-level","title":"Configure Log Level","text":"<p>You can enable the DEBUG log level on <code>gpustack start</code> by setting the <code>--debug</code> parameter.</p> <p>You can configure log level of GPUStack server at runtime by running the following command on the server node:</p> <pre><code>curl -X PUT http://localhost/debug/log_level -d \"debug\"\n</code></pre>"},{"location":"troubleshooting/#reset-admin-password","title":"Reset Admin Password","text":"<p>In case you forgot the admin password, you can reset it by running the following command on the server node:</p> <pre><code>gpustack reset-admin-password\n</code></pre>"},{"location":"upgrade/","title":"Upgrade","text":"<p>You can upgrade GPUStack using the installation script or by manually installing the desired version of the GPUStack Python package.</p> <p>Note</p> <p>When upgrading, upgrade the GPUStack server first, then upgrade the workers.</p>"},{"location":"upgrade/#upgrade-gpustack-using-the-installation-script","title":"Upgrade GPUStack Using the Installation Script","text":"<p>To upgrade GPUStack from an older version, re-run the installation script using the same configuration options you originally used.</p> <p>Running the installation script will:</p> <ol> <li>Install the latest version of the GPUStack Python package.</li> <li>Update the system service (systemd, launchd, or Windows) init script to reflect the arguments passed to the installation script.</li> <li>Restart the GPUStack service.</li> </ol>"},{"location":"upgrade/#linux-and-macos","title":"Linux and macOS","text":"<p>For example, to upgrade GPUStack to the latest version on a Linux system and macOS:</p> <pre><code>curl -sfL https://get.gpustack.ai | &lt;EXISTING_INSTALL_ENV&gt; sh -s - &lt;EXISTING_GPUSTACK_ARGS&gt;\n</code></pre> <p>To upgrade to a specific version, specify the <code>INSTALL_PACKAGE_SPEC</code> environment variable similar to the <code>pip install</code> command:</p> <pre><code>curl -sfL https://get.gpustack.ai | INSTALL_PACKAGE_SPEC=gpustack==x.y.z &lt;EXISTING_INSTALL_ENV&gt; sh -s - &lt;EXISTING_GPUSTACK_ARGS&gt;\n</code></pre>"},{"location":"upgrade/#windows","title":"Windows","text":"<p>To upgrade GPUStack to the latest version on a Windows system:</p> <pre><code>$env:&lt;EXISTING_INSTALL_ENV&gt; = &lt;EXISTING_INSTALL_ENV_VALUE&gt;\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>To upgrade to a specific version:</p> <pre><code>$env:INSTALL_PACKAGE_SPEC = gpustack==x.y.z\n$env:&lt;EXISTING_INSTALL_ENV&gt; = &lt;EXISTING_INSTALL_ENV_VALUE&gt;\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } &lt;EXISTING_GPUSTACK_ARGS&gt;\"\n</code></pre>"},{"location":"upgrade/#docker-upgrade","title":"Docker Upgrade","text":"<p>If you installed GPUStack using Docker, upgrade to the a new version by pulling the Docker image with the desired version tag.</p> <p>For example:</p> <pre><code>docker pull gpustack/gpustack:vX.Y.Z\n</code></pre> <p>Then restart the GPUStack service with the new image.</p>"},{"location":"upgrade/#manual-upgrade","title":"Manual Upgrade","text":"<p>If you install GPUStack manually, upgrade using the common <code>pip</code> workflow.</p> <p>For example, to upgrade GPUStack to the latest version:</p> <pre><code>pip install --upgrade gpustack\n</code></pre> <p>Then restart the GPUStack service according to your setup.</p>"},{"location":"cli-reference/chat/","title":"gpustack chat","text":"<p>Chat with a large language model.</p> <pre><code>gpustack chat model [prompt]\n</code></pre>"},{"location":"cli-reference/chat/#positional-arguments","title":"Positional Arguments","text":"Name Description model The model to use for chat. prompt The prompt to send to the model. [Optional]"},{"location":"cli-reference/chat/#one-time-chat-with-a-prompt","title":"One-time Chat with a Prompt","text":"<p>If a prompt is provided, it performs a one-time inference. For example:</p> <pre><code>gpustack chat llama3 \"tell me a joke.\"\n</code></pre> <p>Example output:</p> <pre><code>Why couldn't the bicycle stand up by itself?\n\nBecause it was two-tired!\n</code></pre>"},{"location":"cli-reference/chat/#interactive-chat","title":"Interactive Chat","text":"<p>If the <code>prompt</code> argument is not provided, you can chat with the large language model interactively. For example:</p> <pre><code>gpustack chat llama3\n</code></pre> <p>Example output:</p> <pre><code>&gt;tell me a joke.\nHere's one:\n\nWhy couldn't the bicycle stand up by itself?\n\n(wait for it...)\n\nBecause it was two-tired!\n\nHope that made you smile!\n&gt;Do you have a better one?\nHere's another one:\n\nWhy did the scarecrow win an award?\n\n(think about it for a sec...)\n\nBecause he was outstanding in his field!\n\nHope that one stuck with you!\n\nDo you want to hear another one?\n&gt;\\quit\n</code></pre>"},{"location":"cli-reference/chat/#interactive-commands","title":"Interactive Commands","text":"<p>Followings are available commands in interactive chat:</p> <pre><code>Commands:\n  \\q or \\quit - Quit the chat\n  \\c or \\clear - Clear chat context in prompt\n  \\? or \\h or \\help - Print this help message\n</code></pre>"},{"location":"cli-reference/chat/#connect-to-external-gpustack-server","title":"Connect to External GPUStack Server","text":"<p>If you are not running <code>gpustack chat</code> on the server node, or if you are serving on a custom host or port, you should provide the following environment variables:</p> Name Description GPUSTACK_SERVER_URL URL of the GPUStack server, e.g., <code>http://myserver</code>. GPUSTACK_API_KEY GPUStack API key."},{"location":"cli-reference/download-tools/","title":"gpustack download-tools","text":"<p>Download dependency tools, including llama-box, gguf-parser, and fastfetch.</p> <pre><code>gpustack download-tools [OPTIONS]\n</code></pre>"},{"location":"cli-reference/download-tools/#configurations","title":"Configurations","text":"Flag Default Description <code>----tools-download-base-url</code> value (empty) Base URL to download dependency tools. <code>--save-archive</code> value (empty) Path to save downloaded tools as a tar archive. <code>--load-archive</code> value (empty) Path to load downloaded tools from a tar archive, instead of downloading. <code>--system</code> value Default is the current OS. Operating system to download tools for. Options: <code>linux</code>, <code>windows</code>, <code>macos</code>. <code>--arch</code> value Default is the current architecture. Architecture to download tools for. Options: <code>amd64</code>, <code>arm64</code>. <code>--device</code> value Default is the current device. Device to download tools for. Options: <code>cuda</code>, <code>mps</code>, <code>npu</code>, <code>musa</code>, <code>cpu</code>."},{"location":"cli-reference/draw/","title":"gpustack draw","text":"<p>Generate an image with a diffusion model.</p> <pre><code>gpustack draw [model] [prompt]\n</code></pre>"},{"location":"cli-reference/draw/#positional-arguments","title":"Positional Arguments","text":"Name Description model The model to use for image generation. prompt Text prompt to use for image generation. <p>The <code>model</code> can be either of the following:</p> <ol> <li>Name of a GPUStack model. You need to create a model in GPUStack before using it here.</li> <li>Reference to a Hugging Face GGUF diffusion model in Ollama style. When using this option, the model will be deployed if it is not already available. When not specified the default <code>Q4_0</code> tag is used. Examples:</li> </ol> <ul> <li><code>hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF</code></li> <li><code>hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF:FP16</code></li> <li><code>hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF:stable-diffusion-v3-5-large-turbo-Q4_0.gguf</code></li> </ul>"},{"location":"cli-reference/draw/#configurations","title":"Configurations","text":"Flag Default Description <code>--size</code> value <code>512x512</code> Size of the image to generate, specified as <code>widthxheight</code>. <code>--sampler</code> value <code>euler</code> Sampling method. Options include: euler_a, euler, heun, dpm2, dpm++2s_a, dpm++2m, lcm, etc. <code>--sample-steps</code> value (Empty) Number of sampling steps. <code>--cfg-scale</code> value (Empty) Classifier-free guidance scale for balancing prompt adherence and creativity. <code>--seed</code> value (Empty) Seed for random number generation. Useful for reproducibility. <code>--negative-prompt</code> value (Empty) Text prompt for what to avoid in the image. <code>--output</code> value (Empty) Path to save the generated image. <code>--show</code> <code>False</code> If True, opens the generated image in the default image viewer. <code>-d</code>, <code>--debug</code> <code>False</code> Enable debug mode."},{"location":"cli-reference/start/","title":"gpustack start","text":"<p>Run GPUStack server or worker.</p> <pre><code>gpustack start [OPTIONS]\n</code></pre>"},{"location":"cli-reference/start/#configurations","title":"Configurations","text":""},{"location":"cli-reference/start/#common-options","title":"Common Options","text":"Flag Default Description <code>--config-file</code> value (empty) Path to the YAML config file. <code>-d</code> value, <code>--debug</code> value <code>False</code> To enable debug mode, the short flag -d is not supported in Windows because this flag is reserved by PowerShell for CommonParameters. <code>--data-dir</code> value (empty) Directory to store data. Default is OS specific. <code>--cache-dir</code> value (empty) Directory to store cache (e.g., model files). Defaults to /cache. <code>-t</code> value, <code>--token</code> value Auto-generated. Shared secret used to add a worker. <code>--huggingface-token</code> value (empty) User Access Token to authenticate to the Hugging Face Hub. Can also be configured via the <code>HF_TOKEN</code> environment variable."},{"location":"cli-reference/start/#server-options","title":"Server Options","text":"Flag Default Description <code>--host</code> value <code>0.0.0.0</code> Host to bind the server to. <code>--port</code> value <code>80</code> Port to bind the server to. <code>--disable-worker</code> <code>False</code> Disable embedded worker. <code>--bootstrap-password</code> value Auto-generated. Initial password for the default admin user. <code>--database-url</code> value <code>sqlite:///&lt;data-dir&gt;/database.db</code> URL of the database. Example: postgresql://user:password@hostname:port/db_name <code>--ssl-keyfile</code> value (empty) Path to the SSL key file. <code>--ssl-certfile</code> value (empty) Path to the SSL certificate file. <code>--force-auth-localhost</code> <code>False</code> Force authentication for requests originating from localhost (127.0.0.1).When set to True, all requests from localhost will require authentication. <code>--ollama-library-base-url</code> <code>https://registry.ollama.ai</code> Base URL for the Ollama library. <code>--disable-update-check</code> <code>False</code> Disable update check."},{"location":"cli-reference/start/#worker-options","title":"Worker Options","text":"Flag Default Description <code>-s</code> value, <code>--server-url</code> value (empty) Server to connect to. <code>--worker-ip</code> value (empty) IP address of the worker node. Auto-detected by default. <code>--disable-metrics</code> <code>False</code> Disable metrics. <code>--disable-rpc-servers</code> <code>False</code> Disable RPC servers. <code>--metrics-port</code> value <code>10151</code> Port to expose metrics. <code>--worker-port</code> value <code>10150</code> Port to bind the worker to. Use a consistent value for all workers. <code>--log-dir</code> value (empty) Directory to store logs. <code>--system-reserved</code> value <code>\"{\\\"ram\\\": 2, \\\"vram\\\": 0}\"</code> The system reserves resources for the worker during scheduling, measured in GiB. By default, 2 GiB of RAM is reserved, Note: '{\\\"memory\\\": 2, \\\"gpu_memory\\\": 0}' is also supported, but it is deprecated and will be removed in future releases. <code>--tools-download-base-url</code> value Base URL for downloading dependency tools."},{"location":"cli-reference/start/#available-environment-variables","title":"Available Environment Variables","text":"<p>Most of the options can be set via environment variables. The environment variables are prefixed with <code>GPUSTACK_</code> and are in uppercase. For example, <code>--data-dir</code> can be set via the <code>GPUSTACK_DATA_DIR</code> environment variable.</p> <p>Below are additional environment variables that can be set:</p> Flag Description <code>HF_ENDPOINT</code> Hugging Face Hub endpoint. e.g., <code>https://hf-mirror.com</code>"},{"location":"cli-reference/start/#config-file","title":"Config File","text":"<p>You can configure start options using a YAML-format config file when starting GPUStack server or worker. Here is a complete example:</p> <pre><code># Common Options\ndebug: false\ndata_dir: /path/to/data_dir\ncache_dir: /path/to/cache_dir\ntoken: mytoken\n\n# Server Options\nhost: 0.0.0.0\nport: 80\ndisable_worker: false\ndatabase_url: postgresql://user:password@hostname:port/db_name\nssl_keyfile: /path/to/keyfile\nssl_certfile: /path/to/certfile\nforce_auth_localhost: false\nbootstrap_password: myadminpassword\nollama_library_base_url: https://registry.mycompany.com\ndisable_update_check: false\n\n# Worker Options\nserver_url: http://myserver\nworker_ip: 192.168.1.101\ndisable_metrics: false\ndisable_rpc_servers: false\nmetrics_port: 10151\nworker_port: 10150\nlog_dir: /path/to/log_dir\nsystem_reserved:\n  ram: 2\n  vram: 0\ntools_download_base_url: https://mirror.mycompany.com\n</code></pre>"},{"location":"installation/air-gapped-installation/","title":"Air-Gapped Installation","text":"<p>You can install GPUStack in an air-gapped environment. An air-gapped environment refers to a setup where GPUStack will be installed offline, behind a firewall, or behind a proxy.</p> <p>The following methods are available for installing GPUStack in an air-gapped environment:</p> <ul> <li>Docker Installation</li> <li>Manual Installation</li> </ul>"},{"location":"installation/air-gapped-installation/#docker-installation","title":"Docker Installation","text":"<p>When running GPUStack with Docker, it works out of the box in an air-gapped environment as long as the Docker images are available. To do this, follow these steps:</p> <ol> <li>Pull GPUStack Docker images in an online environment.</li> <li>Publish Docker images to a private registry.</li> <li>Refer to the Docker Installation guide to run GPUStack using Docker.</li> </ol>"},{"location":"installation/air-gapped-installation/#manual-installation","title":"Manual Installation","text":"<p>For manual installation, you need to prepare the required packages and tools in an online environment and then transfer them to the air-gapped environment.</p>"},{"location":"installation/air-gapped-installation/#prerequisites","title":"Prerequisites","text":"<p>Set up an online environment identical to the air-gapped environment, including OS, architecture, and Python version.</p>"},{"location":"installation/air-gapped-installation/#step-1-download-the-required-packages","title":"Step 1: Download the Required Packages","text":"<p>Run the following commands in an online environment:</p> <pre><code># On Windows (PowerShell):\n# $PACKAGE_SPEC = \"gpustack\"\n\n# Optional: To include extra dependencies (vllm, audio, all) or install a specific version\n# PACKAGE_SPEC=\"gpustack[all]\"\n# PACKAGE_SPEC=\"gpustack==0.4.0\"\nPACKAGE_SPEC=\"gpustack\"\n\n# Download all required packages\npip wheel $PACKAGE_SPEC -w gpustack_offline_packages\n\n# Install GPUStack to access its CLI\npip install gpustack\n\n# Download dependency tools and save them as an archive\ngpustack download-tools --save-archive gpustack_offline_tools.tar.gz\n</code></pre> <p>Optional: Additional Dependencies for macOS.</p> <pre><code># Deploying the speech-to-text CosyVoice model on macOS requires additional dependencies.\nbrew install openfst\nCPLUS_INCLUDE_PATH=$(brew --prefix openfst)/include\nLIBRARY_PATH=$(brew --prefix openfst)/lib\n\nAUDIO_DEPENDENCY_PACKAGE_SPEC=\"wetextprocessing\"\npip wheel $AUDIO_DEPENDENCY_PACKAGE_SPEC -w gpustack_audio_dependency_offline_packages\nmv gpustack_audio_dependency_offline_packages/* gpustack_offline_packages/ &amp;&amp; rm -rf gpustack_audio_dependency_offline_packages\n</code></pre> <p>Note</p> <p>This instruction assumes that the online environment uses the same GPU type as the air-gapped environment. If the GPU types differ, use the <code>--device</code> flag to specify the device type for the air-gapped environment. Refer to the download-tools command for more information.</p>"},{"location":"installation/air-gapped-installation/#step-2-transfer-the-packages","title":"Step 2: Transfer the Packages","text":"<p>Transfer the following files from the online environment to the air-gapped environment.</p> <ul> <li><code>gpustack_offline_packages</code> directory.</li> <li><code>gpustack_offline_tools.tar.gz</code> file.</li> </ul>"},{"location":"installation/air-gapped-installation/#step-3-install-gpustack","title":"Step 3: Install GPUStack","text":"<p>In the air-gapped environment, run the following commands:</p> <pre><code># Install GPUStack from the downloaded packages\npip install --no-index --find-links=gpustack_offline_packages gpustack\n\n# Load and apply the pre-downloaded tools archive\ngpustack download-tools --load-archive gpustack_offline_tools.tar.gz\n</code></pre> <p>Optional: Additional Dependencies for macOS.</p> <pre><code># Install the additional dependencies for speech-to-text CosyVoice model on macOS.\nbrew install openfst\n\npip install --no-index --find-links=gpustack_offline_packages wetextprocessing\n</code></pre> <p>Now you can run GPUStack by following the instructions in the Manual Installation guide.</p>"},{"location":"installation/docker-installation/","title":"Docker Installation","text":"<p>You can use the official Docker image to run GPUStack in a container. Installation using docker is supported on:</p> <ul> <li>Linux with Nvidia GPUs</li> </ul>"},{"location":"installation/docker-installation/#prerequisites","title":"Prerequisites","text":"<ul> <li>Docker</li> <li>Nvidia Container Toolkit</li> </ul>"},{"location":"installation/docker-installation/#run-gpustack-with-docker","title":"Run GPUStack with Docker","text":"<p>Run the following command to start the GPUStack server:</p> <pre><code>docker run -d --gpus all -p 80:80 --ipc=host \\\n    -v gpustack-data:/var/lib/gpustack gpustack/gpustack\n</code></pre> <p>Note</p> <p>You can either use the <code>--ipc=host</code> flag or <code>--shm-size</code> flag to allow the container to access the host\u2019s shared memory. It is used by vLLM and pyTorch to share data between processes under the hood, particularly for tensor parallel inference.</p> <p>You can set additional flags for the <code>gpustack start</code> command by appending them to the docker run command.</p> <p>For example, to start a GPUStack worker:</p> <pre><code>docker run -d --gpus all --ipc=host --network=host \\\n    gpustack/gpustack --server-url http://myserver --token mytoken\n</code></pre> <p>Note</p> <p>The <code>--network=host</code> flag is used to ensure that server is accessible to the worker and inference services running on it. Alternatively, you can set <code>--worker-ip &lt;host-ip&gt; -p 10150:10150 -p 40000-41024:40000-41024</code> to expose relevant ports.</p> <p>For configuration details, please refer to the CLI Reference.</p>"},{"location":"installation/docker-installation/#run-gpustack-with-docker-compose","title":"Run GPUStack with Docker Compose","text":"<p>Get the docker-compose file from GPUStack repository, run the following command to start the GPUStack server:</p> <pre><code>docker-compose up -d\n</code></pre> <p>You can update the <code>docker-compose.yml</code> file to customize the command while starting a GPUStack worker.</p>"},{"location":"installation/docker-installation/#build-your-own-docker-image","title":"Build Your Own Docker Image","text":"<p>The official Docker image is built with CUDA 12.4. If you want to use a different version of CUDA, you can build your own Docker image.</p> <pre><code># Example Dockerfile\nARG CUDA_VERSION=12.4.1\n\nFROM nvidia/cuda:$CUDA_VERSION-cudnn-runtime-ubuntu22.04\n\nENV DEBIAN_FRONTEND=noninteractive\n\nRUN apt-get update &amp;&amp; apt-get install -y \\\n    wget \\\n    tzdata \\\n    python3 \\\n    python3-pip \\\n    &amp;&amp; rm -rf /var/lib/apt/lists/*\n\n\nRUN pip3 install gpustack[all] &amp;&amp; \\\n    pip3 cache purge\n\nENTRYPOINT [ \"gpustack\", \"start\" ]\n</code></pre> <p>Run the following command to build the Docker image:</p> <pre><code>docker build -t my/gpustack --build-arg CUDA_VERSION=12.0.0 .\n</code></pre>"},{"location":"installation/installation-requirements/","title":"Installation Requirements","text":"<p>This page describes the software and networking requirements for the nodes where GPUStack will be installed.</p>"},{"location":"installation/installation-requirements/#python-requirements","title":"Python Requirements","text":"<p>GPUStack requires Python version 3.10 or higher.</p>"},{"location":"installation/installation-requirements/#operating-system-requirements","title":"Operating System Requirements","text":"<p>GPUStack is supported on the following operating systems:</p> <ul> <li> macOS</li> <li> Windows</li> <li> Linux</li> </ul> <p>GPUStack has been tested and verified to work on the following operating systems:</p> OS Versions Windows 10, 11 Ubuntu &gt;= 20.04 Debian &gt;= 11 RHEL &gt;= 8 Rocky &gt;= 8 Fedora &gt;= 36 OpenSUSE &gt;= 15.3 (leap) OpenEuler &gt;= 22.03 <p>Note</p> <p>The installation of GPUStack worker on a Linux system requires that the GLIBC version be 2.29 or higher. If your system uses a lower GLIBC version, consider using the Docker Installation method as an alternative.</p>"},{"location":"installation/installation-requirements/#supported-architectures","title":"Supported Architectures","text":"<p>GPUStack supports both AMD64 and ARM64 architectures, with the following notes:</p> <ul> <li>On Linux and macOS, when using Python versions below 3.12, ensure that the installed Python distribution corresponds to your system architecture.</li> <li>On Windows, please use the AMD64 distribution of Python, as wheel packages for certain dependencies are unavailable for ARM64. If you use tools like <code>conda</code>, this will be handled automatically, as conda installs the AMD64 distribution by default.</li> </ul>"},{"location":"installation/installation-requirements/#accelerator-runtime-requirements","title":"Accelerator Runtime Requirements","text":"<p>GPUStack supports the following accelerators:</p> <ul> <li> Apple Metal (M-series chips)</li> <li> NVIDIA CUDA (Compute Capability 6.0 and above)</li> <li> Ascend CANN</li> <li> Moore Threads MUSA</li> </ul> <p>Ensure all necessary drivers and libraries are installed on the system prior to installing GPUStack.</p>"},{"location":"installation/installation-requirements/#nvidia-cuda","title":"NVIDIA CUDA","text":"<p>To use NVIDIA CUDA as an accelerator, ensure the following components are installed:</p> <ul> <li>NVIDIA CUDA Toolkit</li> <li>NVIDIA cuBLAS (Optional, required for audio models)</li> <li>NVIDIA cuDNN (Optional, required for audio models)</li> <li>NVIDIA Container Toolkit (Optional, required for docker installation)</li> </ul>"},{"location":"installation/installation-requirements/#ascend-cann","title":"Ascend CANN","text":"<p>For Ascend CANN as an accelerator, ensure the following components are installed:</p> <ul> <li>Ascend NPU driver &amp; firmware</li> <li>Ascend CANN Toolkit &amp; kernels</li> </ul>"},{"location":"installation/installation-requirements/#musa","title":"MUSA","text":"<p>To use Moore Threads MUSA as an accelerator, ensure the following components are installed:</p> <ul> <li>MUSA SDK</li> <li>MT Container Toolkits(Optional, required for docker installation)</li> </ul>"},{"location":"installation/installation-requirements/#networking-requirements","title":"Networking Requirements","text":""},{"location":"installation/installation-requirements/#connectivity-requirements","title":"Connectivity Requirements","text":"<p>The following network connectivity is required to ensure GPUStack functions properly:</p> <p>Server-to-Worker: The server must be able to reach the workers for proxying inference requests.</p> <p>Worker-to-Server: Workers must be able to reach the server to register themselves and send updates.</p> <p>Worker-to-Worker: Necessary for distributed inference across multiple workers</p>"},{"location":"installation/installation-requirements/#port-requirements","title":"Port Requirements","text":"<p>GPUStack uses the following ports for communication:</p> <p>Server Ports</p> Port Description TCP 80 Default port for the GPUStack UI and API endpoints TCP 443 Default port for the GPUStack UI and API endpoints (when TLS is enabled) <p>Worker Ports</p> Port Description TCP 10150 Default port for the GPUStack worker TCP 10151 Default port for exposing metrics TCP 40000-41024 Port range allocated for inference services"},{"location":"installation/installation-script/","title":"Installation Script","text":""},{"location":"installation/installation-script/#linux-and-macos","title":"Linux and macOS","text":"<p>You can use the installation script available at <code>https://get.gpustack.ai</code> to install GPUStack as a service on systemd and launchd based systems.</p> <p>You can set additional environment variables and CLI flags when running the script. The following are examples running the installation script with different configurations:</p> <pre><code># Run server.\ncurl -sfL https://get.gpustack.ai | sh -s -\n\n# Run server without the embedded worker.\ncurl -sfL https://get.gpustack.ai | sh -s - --disable-worker\n\n# Run server with TLS.\ncurl -sfL https://get.gpustack.ai | sh -s - --ssl-keyfile /path/to/keyfile --ssl-certfile /path/to/certfile\n\n# Run server with external postgresql database.\ncurl -sfL https://get.gpustack.ai | sh -s - --database-url \"postgresql://username:password@host:port/database_name\"\n\n# Run worker with specified IP.\ncurl -sfL https://get.gpustack.ai | sh -s - --server-url http://myserver --token mytoken --worker-ip 192.168.1.100\n\n# Install with a custom index URL.\ncurl -sfL https://get.gpustack.ai | INSTALL_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple sh -s -\n\n# Install a custom wheel package other than releases form pypi.org.\ncurl -sfL https://get.gpustack.ai | INSTALL_PACKAGE_SPEC=https://repo.mycompany.com/my-gpustack.whl sh -s -\n\n# Install a specific version with extra audio dependencies.\ncurl -sfL https://get.gpustack.ai | INSTALL_PACKAGE_SPEC=gpustack[audio]==0.4.0 sh -s -\n</code></pre>"},{"location":"installation/installation-script/#windows","title":"Windows","text":"<p>You can use the installation script available at <code>https://get.gpustack.ai</code> to install GPUStack as a service on Windows Service Manager.</p> <p>You can set additional environment variables and CLI flags when running the script. The following are examples running the installation script with different configurations:</p> <pre><code># Run server.\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n\n# Run server without the embedded worker.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --disable-worker\"\n\n# Run server with TLS.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --ssl-keyfile 'C:\\path\\to\\keyfile' --ssl-certfile 'C:\\path\\to\\certfile'\"\n\n\n# Run server with external postgresql database.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --database-url 'postgresql://username:password@host:port/database_name'\"\n\n# Run worker with specified IP.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --server-url 'http://myserver' --token 'mytoken' --worker-ip '192.168.1.100'\"\n\n# Run worker with customize reserved resource.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --server-url 'http://myserver' --token 'mytoken' --system-reserved '{\"\"ram\"\":5, \"\"vram\"\":5}'\"\n\n# Install with a custom index URL.\n$env:INSTALL_INDEX_URL = \"https://pypi.tuna.tsinghua.edu.cn/simple\"\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n\n# Install a custom wheel package other than releases form pypi.org.\n$env:INSTALL_PACKAGE_SPEC = \"https://repo.mycompany.com/my-gpustack.whl\"\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n\n# Install a specific version with extra audio dependencies.\n$env:INSTALL_PACKAGE_SPEC = \"gpustack[audio]==0.4.0\"\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>Warning</p> <p>Avoid using PowerShell ISE as it is not compatible with the installation script.</p>"},{"location":"installation/installation-script/#available-environment-variables-for-the-installation-script","title":"Available Environment Variables for the Installation Script","text":"Name Default Description <code>INSTALL_INDEX_URL</code> (empty) Base URL of the Python Package Index. <code>INSTALL_PACKAGE_SPEC</code> <code>gpustack[all]</code> or <code>gpustack[audio]</code> The package spec to install. The install script will automatically decide based on the platform. It supports PYPI package names, URLs, and local paths. See the pip install documentation for details. <ul><li><code>gpustack[all]</code>: With all inference backends: llama-box, vllm, vox-box.</li><li><code>gpustack[vllm]</code>: With inference backends: llama-box, vllm.</li><li><code>gpustack[audio]</code>: With inference backends: llama-box, vox-box.</li></ul> <code>INSTALL_SKIP_POST_CHECK</code> (empty) If set to 1, the installation script will skip the post-installation check."},{"location":"installation/installation-script/#set-environment-variables-for-the-gpustack-service","title":"Set Environment Variables for the GPUStack Service","text":"<p>You can set environment variables for the GPUStack service in an environment file located at:</p> <ul> <li>Linux and macOS: <code>/etc/default/gpustack</code></li> <li>Windows: <code>$env:APPDATA\\gpustack\\gpustack.env</code></li> </ul> <p>The following is an example of the content of the file:</p> <pre><code>HF_TOKEN=\"mytoken\"\nHF_ENDPOINT=\"https://my-hf-endpoint\"\n</code></pre> <p>Note</p> <p>Unlike Systemd, Launchd and Windows services do not natively support reading environment variables from a file. Configuration via the environment file is implemented by the installation script. It reads the file and applies the variables to the service configuration. After modifying the environment file on Windows and macOS, you need to re-run the installation script to apply changes to the GPUStack service.</p>"},{"location":"installation/installation-script/#available-cli-flags","title":"Available CLI Flags","text":"<p>The appended CLI flags of the installation script are passed directly as flags for the <code>gpustack start</code> command. You can refer to the CLI Reference for details.</p>"},{"location":"installation/installation-script/#install-server","title":"Install Server","text":"<p>To set up the GPUStack server (the management node), install GPUStack without the <code>--server-url</code> flag. By default, the GPUStack server includes an embedded worker. To disable this embedded worker on the server, use the <code>--disable-worker</code> flag.</p>"},{"location":"installation/installation-script/#install-worker","title":"Install Worker","text":"<p>To form a cluster, you can add GPUStack workers on additional nodes. Install GPUStack with the <code>--server-url</code> flag to specify the server' address and the <code>--token</code> flag for worker authenticate.</p> <p>Examples are as follows:</p>"},{"location":"installation/installation-script/#linux-or-macos","title":"Linux or macOS","text":"<pre><code>curl -sfL https://get.gpustack.ai | sh -s - --server-url http://myserver --token mytoken\n</code></pre> <p>In the default setup, you can run the following on the server node to get the token used for adding workers:</p> <pre><code>cat /var/lib/gpustack/token\n</code></pre>"},{"location":"installation/installation-script/#windows_1","title":"Windows","text":"<pre><code>Invoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --server-url http://myserver --token mytoken\"\n</code></pre> <p>In the default setup, you can run the following on the server node to get the token used for adding workers:</p> <pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\token\" -Raw\n</code></pre>"},{"location":"installation/manual-installation/","title":"Manual Installation","text":""},{"location":"installation/manual-installation/#prerequites","title":"Prerequites:","text":"<p>Install python3.10 or above with pip.</p>"},{"location":"installation/manual-installation/#install-gpustack-cli","title":"Install GPUStack CLI","text":"<p>Run the following to install GPUStack:</p> <pre><code># You can add extra dependencies, options are \"vllm\", \"audio\" and \"all\".\n# e.g., gpustack[all]\npip install gpustack\n</code></pre> <p>To verify, run:</p> <pre><code>gpustack version\n</code></pre>"},{"location":"installation/manual-installation/#run-gpustack","title":"Run GPUStack","text":"<p>Run the following command to start the GPUStack server:</p> <pre><code>gpustack start\n</code></pre> <p>By default, GPUStack uses <code>/var/lib/gpustack</code> as the data directory so you need <code>sudo</code> or proper permission for that. You can also set a custom data directory by running:</p> <pre><code>gpustack start --data-dir mypath\n</code></pre>"},{"location":"installation/manual-installation/#run-gpustack-as-a-system-service","title":"Run GPUStack as a System Service","text":"<p>A recommended way is to run GPUStack as a startup service. For example, using systemd:</p> <p>Create a service file in <code>/etc/systemd/system/gpustack.service</code>:</p> <pre><code>[Unit]\nDescription=GPUStack Service\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nEnvironmentFile=-/etc/default/%N\nExecStart=gpustack start\nRestart=always\nRestartSec=3\nStandardOutput=append:/var/log/gpustack.log\nStandardError=append:/var/log/gpustack.log\n\n[Install]\nWantedBy=multi-user.target\n</code></pre> <p>Then start GPUStack:</p> <pre><code>systemctl daemon-reload\nsystemctl enable gpustack\n</code></pre>"},{"location":"installation/uninstallation/","title":"Uninstallation","text":""},{"location":"installation/uninstallation/#uninstallation-script","title":"Uninstallation Script","text":"<p>Warning</p> <p>Uninstallation script deletes the data in local datastore(sqlite), configuration, model cache, and all of the scripts and CLI tools. It does not remove any data from external datastores.</p> <p>If you installed GPUStack using the installation script, a script to uninstall GPUStack was generated during installation.</p>"},{"location":"installation/uninstallation/#linux-or-macos","title":"Linux or macOS","text":"<p>Run the following command to uninstall GPUStack:</p> <pre><code>sudo /var/lib/gpustack/uninstall.sh\n</code></pre>"},{"location":"installation/uninstallation/#windows","title":"Windows","text":"<p>Run the following command in PowerShell to uninstall GPUStack:</p> <pre><code>Set-ExecutionPolicy Bypass -Scope Process -Force; &amp; \"$env:APPDATA\\gpustack\\uninstall.ps1\"\n</code></pre>"},{"location":"installation/uninstallation/#manual-uninstallation","title":"Manual Uninstallation","text":"<p>If you install GPUStack manually, the followings are example commands to uninstall GPUStack. You can modify according to your setup:</p> <pre><code># Stop and remove the service.\nsystemctl stop gpustack.service\nrm /etc/systemd/system/gpustack.service\nsystemctl daemon-reload\n# Uninstall the CLI.\npip uninstall gpustack\n# Remove the data directory.\nrm -rf /var/lib/gpustack\n</code></pre>"},{"location":"tutorials/creating-text-embeddings/","title":"Creating Text Embeddings","text":"<p>Text embeddings are numerical representations of text that capture semantic meaning, enabling machines to understand relationships and similarities between different pieces of text. In essence, they transform text into vectors in a continuous space, where texts with similar meanings are positioned closer together. Text embeddings are widely used in applications such as natural language processing, information retrieval, and recommendation systems.</p> <p>In this tutorial, we will demonstrate how to deploy embedding models in GPUStack and generate text embeddings using the deployed models.</p>"},{"location":"tutorials/creating-text-embeddings/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul>"},{"location":"tutorials/creating-text-embeddings/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Enable the <code>GGUF</code> checkbox to filter models by GGUF format.</li> <li>Use the search bar in the top left to search for the model name <code>CompendiumLabs/bge-small-en-v1.5-gguf</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/creating-text-embeddings/#step-2-generate-an-api-key","title":"Step 2: Generate an API Key","text":"<p>We will use the GPUStack API to generate text embeddings, and an API key is required:</p> <ol> <li>Navigate to the <code>API Keys</code> page in the GPUStack UI.</li> <li>Click the <code>New API Key</code> button.</li> <li>Enter a name for the API key and click the <code>Save</code> button.</li> <li>Copy the generated API key. You can only view the API key once, so make sure to save it securely.</li> </ol>"},{"location":"tutorials/creating-text-embeddings/#step-3-generate-text-embeddings","title":"Step 3: Generate Text Embeddings","text":"<p>With the model deployed and an API key, you can generate text embeddings via the GPUStack API. Here is an example script using <code>curl</code>:</p> <pre><code>export SERVER_URL=&lt;your-server-url&gt;\nexport GPUSTACK_API_KEY=&lt;your-api-key&gt;\ncurl $SERVER_URL/v1-openai/embeddings \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"input\": \"The food was delicious and the waiter...\",\n    \"model\": \"bge-small-en-v1.5\",\n    \"encoding_format\": \"float\"\n  }'\n</code></pre> <p>Replace <code>&lt;your-server-url&gt;</code> with the URL of your GPUStack server and <code>&lt;your-api-key&gt;</code> with the API key you generated in the previous step.</p> <p>Example response:</p> <pre><code>{\n  \"data\": [\n    {\n      \"embedding\": [\n        -0.012189436703920364, 0.016934078186750412, 0.003965042531490326,\n        -0.03453584015369415, -0.07623119652271271, -0.007116147316992283,\n        0.11278388649225235, 0.019714849069714546, 0.010370955802500248,\n        -0.04219457507133484, -0.029902394860982895, 0.01122555136680603,\n        0.022912170737981796, 0.031186765059828758, 0.006303929258137941,\n        # ... additional values\n      ],\n      \"index\": 0,\n      \"object\": \"embedding\"\n    }\n  ],\n  \"model\": \"bge-small-en-v1.5\",\n  \"object\": \"list\",\n  \"usage\": { \"prompt_tokens\": 12, \"total_tokens\": 12 }\n}\n</code></pre>"},{"location":"tutorials/inference-on-cpus/","title":"Inference on CPUs","text":"<p>GPUStack supports inference on CPUs, offering flexibility when GPU resources are limited or when model sizes exceed available GPU memory. The following CPU inference modes are available:</p> <ul> <li>CPU+GPU Hybrid Inference: Enables partial acceleration by offloading portions of large models to the CPU when VRAM capacity is insufficient.</li> <li>Full CPU Inference: Operates entirely on CPU when no GPU resources are available.</li> </ul> <p>Note</p> <p>CPU inference is supported when using the llama-box (llama.cpp) backend.</p> <p>To deploy a model with CPU offloading, enable the <code>Allow CPU Offloading</code> option in the deployment configuration (this setting is enabled by default).</p> <p></p> <p>After deployment, you can view the number of model layers offloaded to the CPU.</p> <p></p>"},{"location":"tutorials/inference-with-function-calling/","title":"Inference with Function Calling","text":"<p>Function calling allows you to connect models to external tools and systems. This is useful for many things such as empowering AI assistants with capabilities, or building deep integrations between your applications and the models.</p> <p>In this tutorial, you\u2019ll learn how to set up and use function calling within GPUStack to extend your AI\u2019s capabilities.</p> <p>Note</p> <ol> <li>Function calling is supported in the vLLM inference backend.</li> <li>Function calling is essentially achieved through prompt engineering, requiring models to be trained with internalized templates to enable this capability. Therefore, not all LLMs support function calling.</li> </ol>"},{"location":"tutorials/inference-with-function-calling/#prerequisites","title":"Prerequisites","text":"<p>Before proceeding, ensure the following:</p> <ul> <li>GPUStack is installed and running.</li> <li>A Linux worker node with a GPU is available. We'll use Qwen2.5-7B-Instruct as the model for this tutorial. The model requires a GPU with at least 18GB VRAM.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul>"},{"location":"tutorials/inference-with-function-calling/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI and click the <code>Deploy Model</code> button. In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar to find the <code>Qwen/Qwen2.5-7B-Instruct</code> model.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button and add the following parameters:</li> </ol> <ul> <li><code>--enable-auto-tool-choice</code></li> <li><code>--tool-call-parser=hermes</code></li> </ul> <ol> <li>Click the <code>Save</code> button to deploy the model.</li> </ol> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p>"},{"location":"tutorials/inference-with-function-calling/#step-2-generate-an-api-key","title":"Step 2: Generate an API Key","text":"<p>We will use the GPUStack API to interact with the model. To do this, you need to generate an API key:</p> <ol> <li>Navigate to the <code>API Keys</code> page in the GPUStack UI.</li> <li>Click the <code>New API Key</code> button.</li> <li>Enter a name for the API key and click the <code>Save</code> button.</li> <li>Copy the generated API key for later use.</li> </ol>"},{"location":"tutorials/inference-with-function-calling/#step-3-do-inference","title":"Step 3: Do Inference","text":"<p>With the model deployed and an API key, you can call the model via the GPUStack API. Here is an example script using <code>curl</code> (replace <code>&lt;your-server-url&gt;</code> with your GPUStack server URL and <code>&lt;your-api-key&gt;</code> with the API key generated in the previous step):</p> <pre><code>export GPUSTACK_SERVER_URL=&lt;your-server-url&gt;\nexport GPUSTACK_API_KEY=&lt;your-api-key&gt;\ncurl $GPUSTACK_SERVER_URL/v1-openai/chat/completions \\\n-H \"Content-Type: application/json\" \\\n-H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n-d '{\n  \"model\": \"qwen2.5-7b-instruct\",\n  \"messages\": [\n    {\n      \"role\": \"user\",\n      \"content\": \"What'\\''s the weather like in Boston today?\"\n    }\n  ],\n  \"tools\": [\n    {\n      \"type\": \"function\",\n      \"function\": {\n        \"name\": \"get_current_weather\",\n        \"description\": \"Get the current weather in a given location\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"location\": {\n              \"type\": \"string\",\n              \"description\": \"The city and state, e.g. San Francisco, CA\"\n            },\n            \"unit\": {\n              \"type\": \"string\",\n              \"enum\": [\"celsius\", \"fahrenheit\"]\n            }\n          },\n          \"required\": [\"location\"]\n        }\n      }\n    }\n  ],\n  \"tool_choice\": \"auto\"\n}'\n</code></pre> <p>Example response:</p> <pre><code>{\n  \"model\": \"qwen2.5-7b-instruct\",\n  \"choices\": [\n    {\n      \"index\": 0,\n      \"message\": {\n        \"role\": \"assistant\",\n        \"content\": null,\n        \"tool_calls\": [\n          {\n            \"id\": \"chatcmpl-tool-b99d32848b324eaea4bac5a5830d00b8\",\n            \"type\": \"function\",\n            \"function\": {\n              \"name\": \"get_current_weather\",\n              \"arguments\": \"{\\\"location\\\": \\\"Boston, MA\\\", \\\"unit\\\": \\\"fahrenheit\\\"}\"\n            }\n          }\n        ]\n      },\n      \"finish_reason\": \"tool_calls\"\n    }\n  ],\n  \"usage\": {\n    \"prompt_tokens\": 212,\n    \"total_tokens\": 242,\n    \"completion_tokens\": 30\n  }\n}\n</code></pre>"},{"location":"tutorials/performing-distributed-inference-across-workers/","title":"Performing Distributed Inference Across Workers","text":"<p>This tutorial will guide you through the process of configuring and running distributed inference across multiple workers using GPUStack. Distributed inference allows you to handle larger language models by distributing the computational workload among multiple workers. This is particularly useful when individual workers do not have sufficient resources, such as VRAM, to run the entire model independently.</p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#prerequisites","title":"Prerequisites","text":"<p>Before proceeding, ensure the following:</p> <ul> <li>GPUStack is installed and running. Refer to the Setting Up a Multi-node GPUStack Cluster tutorial if needed.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul> <p>In this tutorial, we\u2019ll assume a cluster with two nodes, each equipped with an NVIDIA P40 GPU (22GB VRAM), as shown in the following image:</p> <p></p> <p>We aim to run a large language model that requires more VRAM than a single worker can provide. For this tutorial, we\u2019ll use the <code>Qwen/Qwen2.5-72B-Instruct</code> model with the <code>q2_k</code> quantization format. The required resources for running this model can be estimated using the gguf-parser tool:</p> <pre><code>$ gguf-parser --hf-repo Qwen/Qwen2.5-72B-Instruct-GGUF --hf-file qwen2.5-72b-instruct-q2_k-00001-of-00007.gguf --ctx-size=8192 --in-short --skip-architecture --skip-metadata --skip-tokenizer\n\n+--------------------------------------------------------------------------------------+\n| ESTIMATE                                                                             |\n+----------------------------------------------+---------------------------------------+\n|                      RAM                     |                 VRAM 0                |\n+--------------------+------------+------------+----------------+----------+-----------+\n| LAYERS (I + T + O) |     UMA    |   NONUMA   | LAYERS (T + O) |    UMA   |   NONUMA  |\n+--------------------+------------+------------+----------------+----------+-----------+\n|      1 + 0 + 0     | 243.89 MiB | 393.89 MiB |     80 + 1     | 2.50 GiB | 28.92 GiB |\n+--------------------+------------+------------+----------------+----------+-----------+\n</code></pre> <p>From the output, we can see that the estimated VRAM requirement for this model exceeds the 22GB VRAM available on each worker node. Thus, we need to distribute the inference across multiple workers to successfully run the model.</p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<p>Follow these steps to deploy the model from Hugging Face, enabling distributed inference:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Enable the <code>GGUF</code> checkbox to filter models by GGUF format.</li> <li>Use the search bar in the top left to search for the model name <code>Qwen/Qwen2.5-72B-Instruct-GGUF</code>.</li> <li>In the <code>Available Files</code> section, select the <code>q2_k</code> quantization format.</li> <li>Expand the <code>Advanced</code> section and scroll down. Disable the <code>Allow CPU Offloading</code> option and verify that the <code>Allow Distributed Inference Across Workers</code> option is enabled(this is enabled by default). GPUStack will evaluate the available resources in the cluster and run the model in a distributed manner if required.</li> <li>Click the <code>Save</code> button to deploy the model.</li> </ol> <p></p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#step-2-verify-the-model-deployment","title":"Step 2: Verify the Model Deployment","text":"<p>Once the model is deployed, verify the deployment on the <code>Models</code> page, where you can view details about how the model is running across multiple workers.</p> <p></p> <p>You can also check worker and GPU resource usage by navigating to the <code>Resources</code> page.</p> <p></p> <p>Finally, go to the <code>Playground</code> page to interact with the model and verify that everything is functioning correctly.</p> <p></p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#conclusion","title":"Conclusion","text":"<p>Congratulations! You have successfully configured and run distributed inference across multiple workers using GPUStack.</p>"},{"location":"tutorials/running-inference-with-ascend-npus/","title":"Running Inference With Ascend NPUs","text":"<p>GPUStack supports running inference on Ascend NPUs. This tutorial will guide you through the configuration steps.</p>"},{"location":"tutorials/running-inference-with-ascend-npus/#system-and-hardware-support","title":"System and Hardware Support","text":"OS Status Verified Linux Support Ubuntu 20.04 Device Status Verified Ascend 910 Support Ascend 910B"},{"location":"tutorials/running-inference-with-ascend-npus/#setup-steps","title":"Setup Steps","text":""},{"location":"tutorials/running-inference-with-ascend-npus/#install-ascend-packages","title":"Install Ascend packages","text":"<ol> <li>Download Ascend packages</li> </ol> <p>Choose the packages according to your system, hardware and GPUStack is compatible with CANN 8.x from resources download center(links below).</p> <p>Download the driver and firmware from here.</p> Package Name Description Ascend-hdk-{chiptype}-npu-driver{version}_linux-{arch}.run Ascend Driver (run format) Ascend-hdk-{chiptype}-npu-firmware{version}.run Ascend (run format) <p>Download the toolkit and kernels from here.</p> Package Name Description Ascend-cann-toolkit_{version}_linux-{arch}.run CANN Toolkit (run format) Ascend-cann-kernels-{chiptype}{version}_linux-{arch}.run CANN Kernels (run format) <ol> <li>Create the user and group for running</li> </ol> <pre><code>sudo groupadd -g HwHiAiUser\nsudo useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash\nsudo usermod -aG HwHiAiUser $USER\n</code></pre> <ol> <li>Install driver</li> </ol> <pre><code>sudo chmod +x Ascend-hdk-xxx-npu-driver_x.x.x_linux-{arch}.run\n# Driver installation, default installation path: \"/usr/local/Ascend\"\nsudo sh Ascend-hdk-xxx-npu-driver_x.x.x_linux-{arch}.run --full --install-for-all\n</code></pre> <p>If you see the following message, the firmware installation is complete:</p> <pre><code>Driver package installed successfully!\n</code></pre> <ol> <li>Verify successful driver installation</li> </ol> <p>After the driver successful installation, run the <code>npu-smi info</code> command to check if the driver was installed correctly.</p> <pre><code>$npu-smi info\n+------------------------------------------------------------------------------------------------+\n| npu-smi 23.0.1                   Version: 23.0.1                                               |\n+---------------------------+---------------+----------------------------------------------------+\n| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|\n| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |\n+===========================+===============+====================================================+\n| 4     910B3               | OK            | 93.6        40                0    / 0             |\n| 0                         | 0000:01:00.0  | 0           0    / 0          3161 / 65536         |\n+===========================+===============+====================================================+\n+---------------------------+---------------+----------------------------------------------------+\n| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |\n+===========================+===============+====================================================+\n| No running processes found in NPU 4                                                            |\n+===========================+===============+====================================================+\n</code></pre> <ol> <li>Install firmware</li> </ol> <pre><code>sudo chmod +x Ascend-hdk-xxx-npu-firmware_x.x.x.x.X.run\nsudo sh Ascend-hdk-xxx-npu-firmware_x.x.x.x.X.run --full\n</code></pre> <p>If you see the following message, the firmware installation is complete:</p> <pre><code>Firmware package installed successfully!\n</code></pre> <ol> <li>Install toolkit and kernels</li> </ol> <p>As an example for Ubuntu, adapt commands according to your system.</p> <p>Check for dependencies to ensure Python, GCC, and other required tools are installed.</p> <pre><code>gcc --version\ng++ --version\nmake --version\ncmake --version\ndpkg -l zlib1g| grep zlib1g| grep ii\ndpkg -l zlib1g-dev| grep zlib1g-dev| grep ii\ndpkg -l libsqlite3-dev| grep libsqlite3-dev| grep ii\ndpkg -l openssl| grep openssl| grep ii\ndpkg -l libssl-dev| grep libssl-dev| grep ii\ndpkg -l libffi-dev| grep libffi-dev| grep ii\ndpkg -l libbz2-dev| grep libbz2-dev| grep ii\ndpkg -l libxslt1-dev| grep libxslt1-dev| grep ii\ndpkg -l unzip| grep unzip| grep ii\ndpkg -l pciutils| grep pciutils| grep ii\ndpkg -l net-tools| grep net-tools| grep ii\ndpkg -l libblas-dev| grep libblas-dev| grep ii\ndpkg -l gfortran| grep gfortran| grep ii\ndpkg -l libblas3| grep libblas3| grep ii\n</code></pre> <p>If the commands return messages showing missing packages, install them as follows (adjust the command if only specific packages are missing):</p> <pre><code>sudo apt-get install -y gcc g++ make cmake zlib1g zlib1g-dev openssl libsqlite3-dev libssl-dev libffi-dev libbz2-dev libxslt1-dev unzip pciutils net-tools libblas-dev gfortran libblas3\n</code></pre> <p>Install Python dependencies:</p> <pre><code>pip3 install --upgrade pip\npip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions\n</code></pre> <p>Install the toolkit and kernels:</p> <pre><code>chmod +x Ascend-cann-toolkit_{vesion}_linux-{arch}.run\nchmod +x Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run\n\nsh Ascend-cann-toolkit_{vesion}_linux-{arch}.run --install\nsh Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run --install\n</code></pre> <p>Once installation completes, you should see a success message like this:</p> <pre><code>xxx install success\n</code></pre> <ol> <li>Configure environment variables</li> </ol> <pre><code>echo \"source ~/Ascend/ascend-toolkit/set_env.sh\" &gt;&gt; ~/.bashrc\nsource ~/.bashrc\n</code></pre> <p>For more details, refer to the Ascend Documentation.</p>"},{"location":"tutorials/running-inference-with-ascend-npus/#installing-gpustack","title":"Installing GPUStack","text":"<p>Once your environment is ready, you can install GPUStack following the installation guide.</p> <p>Once installed, you should see that GPUStack successfully recognizes the Ascend Device in the resources page.</p> <p></p>"},{"location":"tutorials/running-inference-with-ascend-npus/#running-inference","title":"Running Inference","text":"<p>After installation, you can deploy models and run inference. Refer to the model management for usage details.</p> <p>The Ascend NPU supports inference through the llama-box (llama.cpp) backend. For supported models, see the llama.cpp Ascend NPU model supports.</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/","title":"Running Inference With Moore Threads GPUs","text":"<p>GPUStack supports running inference on Moore Threads GPUs. This tutorial provides a comprehensive guide to configuring your system for optimal performance.</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#system-and-hardware-support","title":"System and Hardware Support","text":"OS Architecture Status Verified Linux x86_64 Support Ubuntu 20.04/22.04 Device Status Verified MTT S80 Support Yes MTT S3000 Support Yes MTT S4000 Support Yes"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#prerequisites","title":"Prerequisites","text":"<p>The following instructions are applicable for <code>Ubuntu 20.04/22.04</code> systems with <code>x86_64</code> architecture.</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#configure-the-container-runtime","title":"Configure the Container Runtime","text":"<p>Follow these links to install and configure the container runtime:</p> <ul> <li>Install Docker: Docker Installation Guide</li> <li>Install the latest drivers for MTT S80/S3000/S4000 (currently rc3.1.0): MUSA SDK Download</li> <li>Install the MT Container Toolkits (currently v1.9.0): MT CloudNative Toolkits Download</li> </ul>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#verify-container-runtime-configuration","title":"Verify Container Runtime Configuration","text":"<p>Ensure the output shows the default runtime as <code>mthreads</code>.</p> <pre><code>$ (cd /usr/bin/musa &amp;&amp; sudo ./docker setup $PWD)\n$ docker info | grep mthreads\n Runtimes: mthreads mthreads-experimental runc\n Default Runtime: mthreads\n</code></pre>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#installing-gpustack","title":"Installing GPUStack","text":"<p>To set up an isolated environment for GPUStack, we recommend using Docker.</p> <pre><code>docker run -d --name gpustack-musa -p 9009:80 --ipc=host -v gpustack-data:/var/lib/gpustack \\\n    gpustack/gpustack:main-musa\n</code></pre> <p>This command will:</p> <ul> <li>Start a container with the GPUStack image.</li> <li>Expose the GPUStack web interface on port <code>9009</code>.</li> <li>Mount the <code>gpustack-data</code> volume to store the GPUStack data.</li> </ul> <p>To check the logs of the running container, use the following command:</p> <pre><code>docker logs -f gpustack-musa\n</code></pre> <p>If the following message appears, the GPUStack container is running successfully:</p> <pre><code>2024-11-15T23:37:46+00:00 - gpustack.server.server - INFO - Serving on 0.0.0.0:80.\n2024-11-15T23:37:46+00:00 - gpustack.worker.worker - INFO - Starting GPUStack worker.\n</code></pre> <p>Once the container is running, access the GPUStack web interface by navigating to <code>http://localhost:9009</code> in your browser.</p> <p>After the initial setup for GPUStack, you should see the following screen:</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#dashboard","title":"Dashboard","text":""},{"location":"tutorials/running-inference-with-moorethreads-gpus/#workers","title":"Workers","text":""},{"location":"tutorials/running-inference-with-moorethreads-gpus/#gpus","title":"GPUs","text":""},{"location":"tutorials/running-inference-with-moorethreads-gpus/#running-inference","title":"Running Inference","text":"<p>After installation, you can deploy models and run inference. Refer to the model management for detailed usage instructions.</p> <p>Moore Threads GPUs support inference through the llama-box (llama.cpp) backend. Most recent models are supported (e.g., llama3.2:1b, llama3.2-vision:11b, qwen2.5:7b, etc.).</p> <p>Use <code>mthreads-gmi</code> to verify if the model is offloaded to the GPU.</p> <pre><code>root@a414c45864ee:/# mthreads-gmi\nSat Nov 16 12:00:16 2024\n---------------------------------------------------------------\n    mthreads-gmi:1.14.0          Driver Version:2.7.0\n---------------------------------------------------------------\nID   Name           |PCIe                |%GPU  Mem\n     Device Type    |Pcie Lane Width     |Temp  MPC Capable\n                                         |      ECC Mode\n+-------------------------------------------------------------+\n0    MTT S80        |00000000:01:00.0    |98%   1339MiB(16384MiB)\n     Physical       |16x(16x)            |56C   YES\n                                         |      N/A\n---------------------------------------------------------------\n\n---------------------------------------------------------------\nProcesses:\nID   PID       Process name                         GPU Memory\n                                                         Usage\n+-------------------------------------------------------------+\n0    120       ...ird_party/bin/llama-box/llama-box       2MiB\n0    2022      ...ird_party/bin/llama-box/llama-box    1333MiB\n---------------------------------------------------------------\n</code></pre>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/","title":"Running Inference on Copilot+ PCs with Snapdragon X","text":"<p>GPUStack supports running on ARM64 Windows, enabling use on Snapdragon X-based Copilot+ PCs.</p> <p>Note</p> <p>Only CPU-based inference is supported on Snapdragon X devices. GPUStack does not currently support GPU or NPU acceleration on this platform.</p>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#prerequisites","title":"Prerequisites","text":"<ul> <li>A Copilot+ PC with Snapdragon X. In this tutorial, we use the Dell XPS 13 9345.</li> <li>Install AMD64 Python 3.10 or above. See details</li> </ul>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#installing-gpustack","title":"Installing GPUStack","text":"<p>Run PowerShell as administrator (avoid using PowerShell ISE), then run the following command to install GPUStack:</p> <pre><code>Invoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>After installation, follow the on-screen instructions to obtain credentials and log in to the GPUStack UI.</p>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#deploying-a-model","title":"Deploying a Model","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button and select <code>Ollama Library</code> from the dropdown.</li> <li>Enter <code>llama3.2</code> in the <code>Name</code> field.</li> <li>Select <code>llama3.2</code> from the <code>Ollama Model</code> dropdown.</li> <li>Click <code>Save</code> to deploy the model.</li> </ol> <p>Once deployed, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#running-inference","title":"Running Inference","text":"<p>Navigate to the <code>Playground</code> page in the GPUStack UI, where you can interact with the deployed model.</p> <p></p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/","title":"Setting Up a Multi-node GPUStack Cluster","text":"<p>This tutorial will guide you through setting up a multi-node GPUStack cluster, where you can distribute your workloads across multiple GPU-enabled nodes. This guide assumes you have basic knowledge of running commands on Linux, macOS, or Windows systems.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#prerequisites","title":"Prerequisites","text":"<p>Before starting, ensure you have the following:</p> <ul> <li>Multiple nodes with supported OS and GPUs for GPUStack installation. View supported platforms and supported accelerators for more information.</li> <li>Nodes are connected to the same network and can communicate with each other.</li> </ul>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-1-install-gpustack-on-the-server-node","title":"Step 1: Install GPUStack on the Server Node","text":"<p>First, you need to install GPUStack on one of the nodes to act as the server node. Follow the instructions below based on your operating system.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#linux-or-macos","title":"Linux or macOS","text":"<p>Run the following command on your server node:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s -\n</code></pre>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#windows","title":"Windows","text":"<p>Run PowerShell as administrator and execute the following command:</p> <pre><code>Invoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>Once GPUStack is installed, you can proceed to configure your cluster by adding worker nodes.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-2-retrieve-the-token-from-the-server-node","title":"Step 2: Retrieve the Token from the Server Node","text":"<p>To add worker nodes to the cluster, you need the token generated by GPUStack on the server node. On the server node, run the following command to get the token:</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#linux-or-macos_1","title":"Linux or macOS","text":"<pre><code>cat /var/lib/gpustack/token\n</code></pre>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#windows_1","title":"Windows","text":"<pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\token\" -Raw\n</code></pre> <p>This token will be required in the next steps to authenticate worker nodes.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-3-add-worker-nodes-to-the-cluster","title":"Step 3: Add Worker Nodes to the Cluster","text":"<p>Now, you will install GPUStack on additional nodes (worker nodes) and connect them to the server node using the token.</p> <p>Linux or macOS Worker Nodes</p> <p>Run the following command on each worker node, replacing http://myserver with the URL of your server node and mytoken with the token retrieved in Step 2:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s - --server-url http://myserver --token mytoken\n</code></pre> <p>Windows Worker Nodes</p> <p>Run PowerShell as administrator on each worker node and use the following command, replacing http://myserver and mytoken with the server URL and token:</p> <pre><code>Invoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } --server-url http://myserver --token mytoken\"\n</code></pre> <p>Once the command is executed, each worker node will connect to the main server and become part of the GPUStack cluster.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-4-verify-the-cluster-setup","title":"Step 4: Verify the Cluster Setup","text":"<p>After adding the worker nodes, you can verify that the cluster is set up correctly by accessing the GPUStack UI.</p> <ol> <li>Open a browser and navigate to <code>http://myserver</code> (replace myserver with the actual server URL).</li> <li>Log in with the default credentials (username <code>admin</code>). To retrieve the default password, run the following command on the server node:</li> </ol> <p>Linux or macOS</p> <pre><code>cat /var/lib/gpustack/initial_admin_password\n</code></pre> <p>Windows</p> <pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\initial_admin_password\" -Raw\n</code></pre> <ol> <li>After logging in, navigate to the <code>Resources</code> page in the UI to see all connected nodes and their GPUs. You should see your worker nodes listed and ready for serving LLMs.</li> </ol>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#conclusion","title":"Conclusion","text":"<p>Congratulations! You've successfully set up a multi-node GPUStack cluster! You can now scale your workloads across multiple nodes, making full use of your available GPUs to handle your tasks efficiently.</p>"},{"location":"tutorials/using-audio-models/","title":"Using Audio Models","text":"<p>GPUStack supports running both speech-to-text and text-to-speech models. Speech-to-text models convert audio inputs in various languages into written text, while text-to-speech models transform written text into natural and expressive speech.</p> <p>In this tutorial, we will walk you through deploying and using speech-to-text and text-to-speech models in GPUStack.</p>"},{"location":"tutorials/using-audio-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>A Linux system with AMD architecture or macOS.</li> <li>Access to Hugging Face for downloading the model files.</li> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> </ul>"},{"location":"tutorials/using-audio-models/#running-speech-to-text-model","title":"Running Speech-to-Text Model","text":""},{"location":"tutorials/using-audio-models/#step-1-deploy-speech-to-text-model","title":"Step 1: Deploy Speech-to-Text Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar in the top left to search for the model name <code>Systran/faster-whisper-medium</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-audio-models/#step-2-interact-with-speech-to-text-model-models","title":"Step 2: Interact with Speech-to-Text Model Models","text":"<ol> <li>Navigate to the <code>Playground</code> &gt; <code>Audio</code> page in the GPUStack UI.</li> <li>Select the <code>Speech to Text</code> Tab.</li> <li>Select the deployed model from the top-right dropdown.</li> <li>Click the <code>Upload</code> button to upload audio file or click the <code>Microphone</code> button to record audio.</li> <li>Click the <code>Generate Text Content</code> button to generate the text.</li> </ol>"},{"location":"tutorials/using-audio-models/#running-text-to-speech-model","title":"Running Text-to-Speech Model","text":""},{"location":"tutorials/using-audio-models/#step-1-deploy-text-to-speech-model","title":"Step 1: Deploy Text-to-Speech Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar in the top left to search for the model name <code>FunAudioLLM/CosyVoice-300M</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-audio-models/#step-2-interact-with-text-to-speech-model-models","title":"Step 2: Interact with Text to Speech Model Models","text":"<ol> <li>Navigate to the <code>Playground</code> &gt; <code>Audio</code> page in the GPUStack UI.</li> <li>Select the <code>Text to Speech</code> Tab.</li> <li>Choose the deployed model from the dropdown menu in the top-right corner. Then, configure the voice and output audio format.</li> <li>Input the text to generate.</li> <li>Click the <code>Submit</code> button to generate the text.</li> </ol>"},{"location":"tutorials/using-image-generation-models/","title":"Using Image Generation Models","text":"<p>GPUStack supports deploying and running state-of-the-art image generation models. These models allow you to generate stunning images from textual descriptions, enabling applications in design, content creation, and more.</p> <p>In this tutorial, we will walk you through deploying and using image generation models in GPUStack.</p>"},{"location":"tutorials/using-image-generation-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>A GPU that has at least 12 GB of VRAM.</li> <li>Access to Hugging Face for downloading the model files.</li> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> </ul>"},{"location":"tutorials/using-image-generation-models/#step-1-deploy-the-stable-diffusion-model","title":"Step 1: Deploy the Stable Diffusion Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar in the top left to search for the model name <code>gpustack/stable-diffusion-v3-5-medium-GGUF</code>.</li> <li>In the <code>Available Files</code> section, select the <code>stable-diffusion-v3-5-medium-Q4_0.gguf</code> file.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-image-generation-models/#step-2-use-the-model-for-image-generation","title":"Step 2: Use the Model for Image Generation","text":"<ol> <li>Navigate to the <code>Playground</code> &gt; <code>Image</code> page in the GPUStack UI.</li> <li>Verify that the deployed model is selected from the top-right <code>Model</code> dropdown.</li> <li>Enter a prompt describing the image you want to generate. For example:</li> </ol> <pre><code>a female character with long, flowing hair that appears to be made of ethereal, swirling patterns resembling the Northern Lights or Aurora Borealis. The background is dominated by deep blues and purples, creating a mysterious and dramatic atmosphere. The character's face is serene, with pale skin and striking features. She wears a dark-colored outfit with subtle patterns. The overall style of the artwork is reminiscent of fantasy or supernatural genres.\n</code></pre> <ol> <li>Select <code>euler</code> in the <code>Sampler</code> dropdown.</li> <li>Set the <code>Sample Steps</code> to <code>20</code>.</li> <li>Click the <code>Submit</code> button to create the image.</li> </ol> <p>The generated image will be displayed in the UI. Your image may look different given the seed and randomness involved in the generation process.</p> <p></p>"},{"location":"tutorials/using-image-generation-models/#conclusion","title":"Conclusion","text":"<p>Congratulations! You\u2019ve successfully deployed and used an image generation model in GPUStack. With this setup, you can generate unique and visually compelling images from textual prompts. Experiment with different prompts and settings to push the boundaries of what\u2019s possible.</p>"},{"location":"tutorials/using-reranker-models/","title":"Using Reranker Models","text":"<p>Reranker models are specialized models designed to improve the ranking of a list of items based on relevance to a given query. They are commonly used in information retrieval and search systems to refine initial search results, prioritizing items that are more likely to meet the user\u2019s intent. Reranker models take the initial document list and reorder items to enhance precision in applications such as search engines, recommendation systems, and question-answering tasks.</p> <p>In this tutorial, we will guide you through deploying and using reranker models in GPUStack.</p>"},{"location":"tutorials/using-reranker-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul>"},{"location":"tutorials/using-reranker-models/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Enable the <code>GGUF</code> checkbox to filter models by GGUF format.</li> <li>Use the search bar in the top left to search for the model name <code>gpustack/bge-reranker-v2-m3-GGUF</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-reranker-models/#step-2-generate-an-api-key","title":"Step 2: Generate an API Key","text":"<p>We will use the GPUStack API to interact with the model. To do this, you need to generate an API key:</p> <ol> <li>Navigate to the <code>API Keys</code> page in the GPUStack UI.</li> <li>Click the <code>New API Key</code> button.</li> <li>Enter a name for the API key and click the <code>Save</code> button.</li> <li>Copy the generated API key. You can only view the API key once, so make sure to save it securely.</li> </ol>"},{"location":"tutorials/using-reranker-models/#step-3-reranking","title":"Step 3: Reranking","text":"<p>With the model deployed and an API key, you can rerank a list of documents via the GPUStack API. Here is an example script using <code>curl</code>:</p> <pre><code>export SERVER_URL=&lt;your-server-url&gt;\nexport GPUSTACK_API_KEY=&lt;your-api-key&gt;\ncurl $SERVER_URL/v1/rerank \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -d '{\n        \"model\": \"bge-reranker-v2-m3\",\n        \"query\": \"What is a panda?\",\n        \"top_n\": 3,\n        \"documents\": [\n            \"hi\",\n            \"it is a bear\",\n            \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n        ]\n    }' | jq\n</code></pre> <p>Replace <code>&lt;your-server-url&gt;</code> with the URL of your GPUStack server and <code>&lt;your-api-key&gt;</code> with the API key you generated in the previous step.</p> <p>Example response:</p> <pre><code>{\n  \"model\": \"bge-reranker-v2-m3\",\n  \"object\": \"list\",\n  \"results\": [\n    {\n      \"document\": {\n        \"text\": \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n      },\n      \"index\": 2,\n      \"relevance_score\": 1.951932668685913\n    },\n    {\n      \"document\": {\n        \"text\": \"it is a bear\"\n      },\n      \"index\": 1,\n      \"relevance_score\": -3.7347371578216553\n    },\n    {\n      \"document\": {\n        \"text\": \"hi\"\n      },\n      \"index\": 0,\n      \"relevance_score\": -6.157620906829834\n    }\n  ],\n  \"usage\": {\n    \"prompt_tokens\": 69,\n    \"total_tokens\": 69\n  }\n}\n</code></pre>"},{"location":"tutorials/using-vision-language-models/","title":"Using Vision Language Models","text":"<p>Vision Language Models can process both visual (image) and language (text) data simultaneously, making them versatile tools for various applications, such as image captioning, visual question answering, and more. In this tutorial, you will learn how to deploy and interact with Vision Language Models (VLMs) in GPUStack.</p> <p>The procedure for deploying and interacting with these models in GPUStack is similar. The main difference is the parameters you need to set when deploying the models. For more information on the parameters you can set, please refer to Backend Parameters .</p> <p>In this tutorial, we will cover the deployment of the following models:</p> <ul> <li>Llama3.2-Vision</li> <li>Qwen2-VL</li> <li>Pixtral</li> <li>Phi3.5-Vision</li> </ul>"},{"location":"tutorials/using-vision-language-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>A Linux machine with one or more GPUs that has at least 30 GB of VRAM in total. We will use the vLLM backend which only supports Linux.</li> <li>Access to Hugging Face and a Hugging Face API key for downloading the model files.</li> <li>You have been granted access to the above models on Hugging Face. Llama3.2-VL and Pixtral are gated models, and you need to request access to them.</li> </ul> <p>Note</p> <p>An Ubuntu node equipped with one H100 (80GB) GPU is used throughout this tutorial.</p>"},{"location":"tutorials/using-vision-language-models/#step-1-install-gpustack","title":"Step 1: Install GPUStack","text":"<p>Run the following command to install GPUStack:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s - --huggingface-token &lt;Hugging Face API Key&gt;\n</code></pre> <p>Replace <code>&lt;Hugging Face API Key&gt;</code> with your Hugging Face API key. GPUStack will use this key to download the model files.</p>"},{"location":"tutorials/using-vision-language-models/#step-2-log-in-to-gpustack-ui","title":"Step 2: Log in to GPUStack UI","text":"<p>Run the following command to get the default password:</p> <pre><code>cat /var/lib/gpustack/initial_admin_password\n</code></pre> <p>Open your browser and navigate to <code>http://&lt;your-server-ip&gt;</code>. Replace <code>&lt;your-server-ip&gt;</code> with the IP address of your server. Log in using the username <code>admin</code> and the password you obtained in the previous step.</p>"},{"location":"tutorials/using-vision-language-models/#step-3-deploy-vision-language-models","title":"Step 3: Deploy Vision Language Models","text":""},{"location":"tutorials/using-vision-language-models/#deploy-llama32-vision","title":"Deploy Llama3.2-Vision","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>meta-llama/Llama-3.2-11B-Vision-Instruct</code> in the search bar.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button multiple times and add the following parameters:</li> </ol> <ul> <li><code>--enforce-eager</code></li> <li><code>--max-num-seqs=16</code></li> <li><code>--max-model-len=8192</code></li> </ul> <ol> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#deploy-qwen2-vl","title":"Deploy Qwen2-VL","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>Qwen/Qwen2-VL-7B-Instruct</code> in the search bar.</li> <li>Click the <code>Save</code> button. The default configurations should work as long as you have enough GPU resources.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#deploy-pixtral","title":"Deploy Pixtral","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>mistralai/Pixtral-12B-2409</code> in the search bar.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button multiple times and add the following parameters:</li> </ol> <ul> <li><code>--tokenizer-mode=mistral</code></li> <li><code>--limit-mm-per-prompt=image=4</code></li> </ul> <ol> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#deploy-phi35-vision","title":"Deploy Phi3.5-Vision","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>microsoft/Phi-3.5-vision-instruct</code> in the search bar.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button and add the following parameter:</li> </ol> <ul> <li><code>--trust-remote-code</code></li> </ul> <ol> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#step-4-interact-with-vision-language-models","title":"Step 4: Interact with Vision Language Models","text":"<ol> <li>Navigate to the <code>Playground</code> page in the GPUStack UI.</li> <li>Select the deployed model from the top-right dropdown.</li> <li>Click on the <code>Upload Image</code> button above the input text area and upload an image.</li> <li>Enter a prompt in the input text area. For example, \"Describe the image.\"</li> <li>Click the <code>Submit</code> button to generate the output.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#conclusion","title":"Conclusion","text":"<p>In this tutorial, you learned how to deploy and interact with Vision Language Models in GPUStack. You can use the same approach to deploy other Vision Language Models not covered in this tutorial. If you have any questions or need further assistance, feel free to reach out to us.</p>"},{"location":"user-guide/api-key-management/","title":"API Key Management","text":"<p>GPUStack supports authentication using API keys. Each GPUStack user can generate and manage their own API keys.</p>"},{"location":"user-guide/api-key-management/#create-api-key","title":"Create API Key","text":"<ol> <li>Navigate to the <code>API Keys</code> page.</li> <li>Click the <code>New API Key</code> button.</li> <li>Fill in the <code>Name</code>, <code>Description</code>, and select the <code>Expiration</code> of the API key.</li> <li>Click the <code>Save</code> button.</li> <li>Copy and store the key somewhere safe, then click the <code>Done</code> button.</li> </ol> <p>Note</p> <p>Please note that you can only see the generated API key once upon creation.</p>"},{"location":"user-guide/api-key-management/#delete-api-key","title":"Delete API Key","text":"<ol> <li>Navigate to the <code>API Keys</code> page.</li> <li>Find the API key you want to delete.</li> <li>Click the <code>Delete</code> button in the <code>Operations</code> column.</li> <li>Confirm the deletion.</li> </ol>"},{"location":"user-guide/api-key-management/#use-api-key","title":"Use API Key","text":"<p>GPUStack supports using the API key as a bearer token. The following is an example using curl:</p> <pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -d '{\n    \"model\": \"llama3\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a helpful assistant.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n      }\n    ],\n    \"stream\": true\n  }'\n</code></pre>"},{"location":"user-guide/image-generation-apis/","title":"Image Generation APIs","text":"<p>GPUStack provides APIs for generating images given a prompt and/or an input image when running diffusion models.</p> <p>Note</p> <p>The image generation APIs are only available when using the llama-box inference backend.</p>"},{"location":"user-guide/image-generation-apis/#supported-models","title":"Supported Models","text":"<p>The following models are available for image generation:</p> <p>Tip</p> <p>Please use the converted GGUF models provided by GPUStack. Check the model link for more details.</p> <ul> <li>stabilityai/stable-diffusion-3.5-large-turbo</li> <li>stabilityai/stable-diffusion-3.5-large</li> <li>stabilityai/stable-diffusion-3.5-medium</li> <li>stabilityai/stable-diffusion-3-medium</li> <li>TencentARC/FLUX.1-mini</li> <li>Freepik/FLUX.1-lite</li> <li>black-forest-labs/FLUX.1-dev</li> <li>black-forest-labs/FLUX.1-schnell</li> <li>stabilityai/sdxl-turbo</li> <li>stabilityai/stable-diffusion-xl-refiner-1.0</li> <li>stabilityai/stable-diffusion-xl-base-1.0</li> <li>stabilityai/sd-turbo</li> <li>stabilityai/stable-diffusion-2-1</li> <li>stable-diffusion-v1-5/stable-diffusion-v1-5</li> <li>CompVis/stable-diffusion-v1-4</li> </ul>"},{"location":"user-guide/image-generation-apis/#api-details","title":"API Details","text":"<p>The image generation APIs adhere to OpenAI API specification. While OpenAI APIs for image generation are simple and opinionated, GPUStack extends these capabilities with additional features.</p>"},{"location":"user-guide/image-generation-apis/#create-image","title":"Create Image","text":""},{"location":"user-guide/image-generation-apis/#streaming","title":"Streaming","text":"<p>This image generation API supports streaming responses to return the progressing of the generation. To enable streaming, set the <code>stream</code> parameter to <code>true</code> in the request body. Example:</p> <pre><code>REQUEST : (application/json)\n{\n  \"n\": 1,\n  \"response_format\": \"b64_json\",\n  \"size\": \"512x512\",\n  \"prompt\": \"A lovely cat\",\n  \"quality\": \"standard\",\n  \"stream\": true,\n  \"stream_options\": {\n    \"include_usage\": true, // return usage information\n  }\n}\n\nRESPONSE : (text/event-stream)\ndata: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\ndata: [DONE]\n</code></pre>"},{"location":"user-guide/image-generation-apis/#advanced-options","title":"Advanced Options","text":"<p>This image generation API supports additional options to control the generation process. The following options are available:</p> <pre><code>REQUEST : (application/json)\n{\n  \"n\": 1,\n  \"response_format\": \"b64_json\",\n  \"size\": \"512x512\",\n  \"prompt\": \"A lovely cat\",\n  \"sampler\": \"euler\",      // required, select from euler_a;euler;heun;dpm2;dpm++2s_a;dpm++2m;dpm++2mv2;ipndm;ipndm_v;lcm\n  \"schedule\": \"default\",   // optional, select from default;discrete;karras;exponential;ays;gits\n  \"seed\": null,            // optional, random seed\n  \"cfg_scale\": 4.5,        // optional, for sampler, the scale of classifier-free guidance in the output phase\n  \"sample_steps\": 20,      // optional, number of sample steps\n  \"negative_prompt\": \"\",   // optional, negative prompt\n  \"stream\": true,\n  \"stream_options\": {\n    \"include_usage\": true, // return usage information\n  }\n}\n\nRESPONSE : (text/event-stream)\ndata: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\ndata: [DONE]\n</code></pre>"},{"location":"user-guide/image-generation-apis/#create-image-edit","title":"Create Image Edit","text":""},{"location":"user-guide/image-generation-apis/#streaming_1","title":"Streaming","text":"<p>This image generation API supports streaming responses to return the progressing of the generation. To enable streaming, set the <code>stream</code> parameter to <code>true</code> in the request body. Example:</p> <pre><code>REQUEST: (multipart/form-data)\nn=1\nresponse_format=b64_json\nsize=512x512\nprompt=\"A lovely cat\"\nquality=standard\nimage=...                         // required\nmask=...                          // optional\nstream=true\nstream_options_include_usage=true // return usage information\n\nRESPONSE : (text/event-stream)\nCASE 1: correct input image\n  data: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\n  data: [DONE]\nCASE 2: illegal input image\n  error: {\"code\": 400, \"message\": \"Invalid image\", \"type\": \"invalid_request_error\"}\n</code></pre>"},{"location":"user-guide/image-generation-apis/#advanced-options_1","title":"Advanced Options","text":"<p>This image generation API supports additional options to control the generation process. The following options are available:</p> <pre><code>REQUEST: (multipart/form-data)\nn=1\nresponse_format=b64_json\nsize=512x512\nprompt=\"A lovely cat\"\nimage=...                         // required\nmask=...                          // optional\nsampler=euler                     // required, select from euler_a;euler;heun;dpm2;dpm++2s_a;dpm++2m;dpm++2mv2;ipndm;ipndm_v;lcm\nschedule=default                  // optional, select from default;discrete;karras;exponential;ays;gits\nseed=null                         // optional, random seed\ncfg_scale=4.5                     // optional, for sampler, the scale of classifier-free guidance in the output phase\nsample_steps=20                   // optional, number of sample steps\nnegative_prompt=\"\"                // optional, negative prompt\nstream=true\nstream_options_include_usage=true // return usage information\n\nRESPONSE : (text/event-stream)\nCASE 1: correct input image\n  data: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\n  data: [DONE]\nCASE 2: illegal input image\n  error: {\"code\": 400, \"message\": \"Invalid image\", \"type\": \"invalid_request_error\"}\n</code></pre>"},{"location":"user-guide/image-generation-apis/#usage","title":"Usage","text":"<p>The followings are examples using the image generation APIs:</p>"},{"location":"user-guide/image-generation-apis/#curl-create-image","title":"curl (Create Image)","text":"<pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/image/generate \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -d '{\n        \"n\": 1,\n        \"response_format\": \"b64_json\",\n        \"size\": \"512x512\",\n        \"prompt\": \"A lovely cat\",\n        \"quality\": \"standard\",\n        \"stream\": true,\n        \"stream_options\": {\n        \"include_usage\": true\n        }\n    }'\n</code></pre>"},{"location":"user-guide/image-generation-apis/#curl-create-image-edit","title":"curl (Create Image Edit)","text":"<pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/image/edit \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -F image=\"@otter.png\" \\\n    -F mask=\"@mask.png\" \\\n    -F prompt=\"A lovely cat\" \\\n    -F n=1 \\\n    -F size=\"512x512\"\n</code></pre>"},{"location":"user-guide/inference-backends/","title":"Inference Backends","text":"<p>GPUStack supports the following inference backends:</p> <ul> <li>llama-box</li> <li>vLLM</li> <li>vox-box</li> </ul> <p>When users deploy a model, the backend is selected automatically based on the following criteria:</p> <ul> <li>If the model is a GGUF model, <code>llama-box</code> is used.</li> <li>If the model is a known <code>text-to-speech</code> or <code>speech-to-text</code> model, <code>vox-box</code> is used.</li> <li>Otherwise, <code>vLLM</code> is used.</li> </ul>"},{"location":"user-guide/inference-backends/#llama-box","title":"llama-box","text":"<p>llama-box is a LM inference server based on llama.cpp and stable-diffusion.cpp.</p>"},{"location":"user-guide/inference-backends/#supported-platforms","title":"Supported Platforms","text":"<p>The llama-box backend supports Linux, macOS and Windows (with CPU offloading only on Windows ARM architecture) platforms.</p>"},{"location":"user-guide/inference-backends/#supported-models","title":"Supported Models","text":"<ul> <li>LLMs: For supported LLMs, refer to the llama.cpp README.</li> <li>Difussion Models: Supported models are listed in this Hugging Face collection.</li> <li>Reranker Models: Supported models can be found in this Hugging Face collection.</li> </ul>"},{"location":"user-guide/inference-backends/#supported-features","title":"Supported Features","text":""},{"location":"user-guide/inference-backends/#allow-cpu-offloading","title":"Allow CPU Offloading","text":"<p>After enabling CPU offloading, GPUStack prioritizes loading as many layers as possible onto the GPU to optimize performance. If GPU resources are limited, some layers will be offloaded to the CPU, with full CPU inference used only when no GPU is available.</p>"},{"location":"user-guide/inference-backends/#allow-distributed-inference-across-workers","title":"Allow Distributed Inference Across Workers","text":"<p>Enable distributed inference across multiple workers. The primary Model Instance will communicate with backend instances on one or more others workers, offloading computation tasks to them.</p>"},{"location":"user-guide/inference-backends/#parameters-reference","title":"Parameters Reference","text":"<p>See the full list of supported parameters for llama-box here.</p>"},{"location":"user-guide/inference-backends/#vllm","title":"vLLM","text":"<p>vLLM is a high-throughput and memory-efficient LLMs inference engine. It is a popular choice for running LLMs in production. vLLM seamlessly supports most state-of-the-art open-source models, including: Transformer-like LLMs (e.g., Llama), Mixture-of-Expert LLMs (e.g., Mixtral), Embedding Models (e.g. E5-Mistral), Multi-modal LLMs (e.g., LLaVA)</p> <p>By default, GPUStack estimates the VRAM requirement for the model instance based on the model's metadata. You can customize the parameters to fit your needs. The following vLLM parameters might be useful:</p> <ul> <li><code>--gpu-memory-utilization</code> (default: 0.9): The fraction of GPU memory to use for the model instance.</li> <li><code>--max-model-len</code>: Model context length. For large-context models, GPUStack automatically sets this parameter to <code>8192</code> to simplify model deployment, especially in resource constrained environments. You can customize this parameter to fit your needs.</li> <li><code>--tensor-parallel-size</code>: Number of tensor parallel replicas. By default, GPUStack sets this parameter given the GPU resources available and the estimation of the model's memory requirement. You can customize this parameter to fit your needs.</li> </ul> <p>For more details, please refer to vLLM documentation.</p>"},{"location":"user-guide/inference-backends/#supported-platforms_1","title":"Supported Platforms","text":"<p>The vLLM backend works on AMD Linux.</p> <p>Note</p> <ol> <li>When users install GPUStack on amd64 Linux using the installation script, vLLM is automatically installed.</li> <li>When users deploy a model using the vLLM backend, GPUStack sets worker label selectors to <code>{\"os\": \"linux\", \"arch\": \"amd64\"}</code> by default to ensure the model instance is scheduled to proper workers. You can customize the worker label selectors in the model configuration.</li> </ol>"},{"location":"user-guide/inference-backends/#supported-models_1","title":"Supported Models","text":"<p>Please refer to the vLLM documentation for supported models.</p>"},{"location":"user-guide/inference-backends/#supported-features_1","title":"Supported Features","text":""},{"location":"user-guide/inference-backends/#multimodal-language-models","title":"Multimodal Language Models","text":"<p>vLLM supports multimodal language models listed here. When users deploy a vision language model using the vLLM backend, image inputs are supported in the chat completion API.</p>"},{"location":"user-guide/inference-backends/#parameters-reference_1","title":"Parameters Reference","text":"<p>See the full list of supported parameters for vLLM here.</p>"},{"location":"user-guide/inference-backends/#vox-box","title":"vox-box","text":"<p>vox-box is an inference engine designed for deploying text-to-speech and speech-to-text models. It also provides an API that is fully compatible with the OpenAI audio API.</p>"},{"location":"user-guide/inference-backends/#supported-platforms_2","title":"Supported Platforms","text":"<p>The vox-box backend supports Linux, macOS and Windows platforms.</p> <p>Note</p> <ol> <li>To use Nvidia GPUs, ensure the following NVIDIA libraries are installed on workers:<ul> <li>cuBLAS for CUDA 12</li> <li>cuDNN 9 for CUDA 12</li> </ul> </li> <li>When users install GPUStack on Linux, macOS and Windows using the installation script, vox-box is automatically installed.</li> <li>CosyVoice models are natively supported on Linux AMD architecture and macOS. However, these models are not supported on Linux ARM or Windows architectures.</li> </ol>"},{"location":"user-guide/inference-backends/#supported-models_2","title":"Supported Models","text":"Model Type Link Supported Platforms Faster-whisper-large-v3 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-large-v2 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-large-v1 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-medium speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-medium.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-small speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-small.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-distil-whisper-large-v3 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-distil-whisper-large-v2 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-distil-whisper-medium.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-tiny speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-tiny.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows CosyVoice-300M-Instruct text-to-speech Hugging Face, ModelScope Linux(ARM not supported), macOS, Windows(Not supported) CosyVoice-300M-SFT text-to-speech Hugging Face, ModelScope Linux(ARM not supported), macOS, Windows(Not supported) CosyVoice-300M text-to-speech Hugging Face, ModelScope Linux(ARM not supported), macOS, Windows(Not supported) CosyVoice-300M-25Hz text-to-speech ModelScope Linux(ARM not supported), macOS, Windows(Not supported)"},{"location":"user-guide/inference-backends/#supported-features_2","title":"Supported Features","text":""},{"location":"user-guide/inference-backends/#allow-gpucpu-offloading","title":"Allow GPU/CPU Offloading","text":"<p>vox-box supports deploying models to NVIDIA GPUs. If GPU resources are insufficient, it will automatically deploy the models to the CPU.</p>"},{"location":"user-guide/model-management/","title":"Model Management","text":"<p>You can manage large language models in GPUStack by navigating to the <code>Models</code> page. A model in GPUStack contains one or multiple replicas of model instances. On deployment, GPUStack automatically computes resource requirements for the model instances from model metadata and schedules them to available workers accordingly.</p>"},{"location":"user-guide/model-management/#deploy-model","title":"Deploy Model","text":"<p>Currently, models from Hugging Face, ModelScope, Ollama and local paths are supported.</p>"},{"location":"user-guide/model-management/#deploying-a-hugging-face-model","title":"Deploying a Hugging Face Model","text":"<ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</p> </li> <li> <p>Search the model by name from Hugging Face using the search bar in the top left. For example, <code>microsoft/Phi-3-mini-4k-instruct-gguf</code>. If you only want to search for GGUF models, check the \"GGUF\" checkbox.</p> </li> <li> <p>Select a file with the desired quantization format from <code>Available Files</code>.</p> </li> <li> <p>Adjust the <code>Name</code> and <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#deploying-a-modelscope-model","title":"Deploying a ModelScope Model","text":"<ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>ModelScope</code> in the dropdown.</p> </li> <li> <p>Search the model by name from ModelScope using the search bar in the top left. For example, <code>Qwen/Qwen2-0.5B-Instruct</code>. If you only want to search for GGUF models, check the \"GGUF\" checkbox.</p> </li> <li> <p>Select a file with the desired quantization format from <code>Available Files</code>.</p> </li> <li> <p>Adjust the <code>Name</code> and <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#deploying-an-ollama-model","title":"Deploying an Ollama Model","text":"<ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>Ollama Library</code> in the dropdown.</p> </li> <li> <p>Fill in the <code>Name</code> of the model.</p> </li> <li> <p>Select an <code>Ollama Model</code> from the dropdown list, or input any Ollama model you need. For example, <code>llama3</code>, <code>llama3:70b</code> or <code>youraccount/llama3:70b</code>.</p> </li> <li> <p>Adjust the <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#deploying-a-local-path-model","title":"Deploying a Local Path Model","text":"<p>You can deploy a model from a local path. The model path can be a directory (e.g., a downloaded Hugging Face model directory) or a file (e.g., a GGUF model file) located on workers. This is useful when running in an air-gapped environment.</p> <p>Note</p> <ol> <li>GPUStack does not check the validity of the model path for scheduling, which may lead to deployment failure if the model path is inaccessible. It is recommended to ensure the model path is accessible on all workers(e.g., using NFS, rsync, etc.). You can also use the worker selector configuration to deploy the model to specific workers.</li> <li>GPUStack cannot evaluate the model's resource requirements unless the server has access to the same model path. Consequently, you may observe empty VRAM/RAM allocations for a deployed model. To mitigate this, it is recommended to make the model files available on the same path on the server. Alternatively, you can customize backend parameters, such as <code>tensor-split</code>, to configure how the model is distributed across the GPUs.</li> </ol> <p>To deploy a local path model:</p> <ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>Local Path</code> in the dropdown.</p> </li> <li> <p>Fill in the <code>Name</code> of the model.</p> </li> <li> <p>Fill in the <code>Model Path</code>.</p> </li> <li> <p>Adjust the <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#edit-model","title":"Edit Model","text":"<ol> <li>Find the model you want to edit on the model list page.</li> <li>Click the <code>Edit</code> button in the <code>Operations</code> column.</li> <li>Update the attributes as needed. For example, change the <code>Replicas</code> to scale up or down.</li> <li>Click the <code>Save</code> button.</li> </ol> <p>Note</p> <p>After editing the model, the configuration will not be applied to existing model instances. You need to delete the existing model instances. GPUStack will recreate new instances based on the updated model configuration.</p>"},{"location":"user-guide/model-management/#delete-model","title":"Delete Model","text":"<ol> <li>Find the model you want to delete on the model list page.</li> <li>Click the ellipsis button in the <code>Operations</code> column, then select <code>Delete</code>.</li> <li>Confirm the deletion.</li> </ol>"},{"location":"user-guide/model-management/#view-model-instance","title":"View Model Instance","text":"<ol> <li>Find the model you want to check on the model list page.</li> <li>Click the <code>&gt;</code> symbol to view the instance list of the model.</li> </ol>"},{"location":"user-guide/model-management/#delete-model-instance","title":"Delete Model Instance","text":"<ol> <li>Find the model you want to check on the model list page.</li> <li>Click the <code>&gt;</code> symbol to view the instance list of the model.</li> <li>Find the model instance you want to delete.</li> <li>Click the ellipsis button for the model instance in the <code>Operations</code> column, then select <code>Delete</code>.</li> <li>Confirm the deletion.</li> </ol> <p>Note</p> <p>After a model instance is deleted, GPUStack will recreate a new instance to satisfy the expected replicas of the model if necessary.</p>"},{"location":"user-guide/model-management/#view-model-instance-logs","title":"View Model Instance Logs","text":"<ol> <li>Find the model you want to check on the model list page.</li> <li>Click the <code>&gt;</code> symbol to view the instance list of the model.</li> <li>Find the model instance you want to check.</li> <li>Click the <code>View Logs</code> button for the model instance in the <code>Operations</code> column.</li> </ol>"},{"location":"user-guide/model-management/#use-self-hosted-ollama-models","title":"Use Self-hosted Ollama Models","text":"<p>You can deploy self-hosted Ollama models by configuring the <code>--ollama-library-base-url</code> option in the GPUStack server. The <code>Ollama Library</code> URL should point to the base URL of the Ollama model registry. For example, <code>https://registry.mycompany.com</code>.</p> <p>Here is an example workflow to set up a registry, publish a model, and use it in GPUStack:</p> <pre><code># Run a self-hosted OCI registry\ndocker run -d -p 5001:5000 --name registry registry:2\n\n# Push a model to the registry using Ollama\nollama pull llama3\nollama cp llama3 localhost:5001/library/llama3\nollama push localhost:5001/library/llama3 --insecure\n\n# Start GPUStack server with the custom Ollama library URL\ncurl -sfL https://get.gpustack.ai | sh -s - --ollama-library-base-url http://localhost:5001\n</code></pre> <p>That's it! You can now deploy the model <code>llama3</code> from <code>Ollama Library</code> source in GPUStack as usual, but the model will now be fetched from the self-hosted registry.</p>"},{"location":"user-guide/model-management/#advanced-model-configuration","title":"Advanced Model Configuration","text":"<p>GPUStack supports tailored configurations for model deployment.</p>"},{"location":"user-guide/model-management/#schedule-type","title":"Schedule Type","text":""},{"location":"user-guide/model-management/#auto","title":"Auto","text":"<p>GPUStack automatically schedules model instances to appropriate GPUs/Workers based on current resource availability.</p> <ul> <li> <p>Placement Strategy</p> </li> <li> <p>Spread: Make the resources of the entire cluster relatively evenly distributed among all workers. It may produce more resource fragmentation on a single worker.</p> </li> <li> <p>Binpack: Prioritize the overall utilization of cluster resources, reducing resource fragmentation on Workers/GPUs.</p> </li> <li> <p>Worker Selector</p> </li> </ul> <p>When configured, the scheduler will deploy the model instance to the worker containing specified labels.</p> <ol> <li> <p>Navigate to the <code>Resources</code> page and edit the desired worker. Assign custom labels to the worker by adding them in the labels section.</p> </li> <li> <p>Go to the <code>Models</code> page and click on the <code>Deploy Model</code> button. Expand the <code>Advanced</code> section and input the previously assigned worker labels in the <code>Worker Selector</code> configuration. During deployment, the Model Instance will be allocated to the corresponding worker based on these labels.</p> </li> </ol>"},{"location":"user-guide/model-management/#manual","title":"Manual","text":"<p>This schedule type allows users to specify which GPU to deploy the model instance on.</p> <ul> <li>GPU Selector</li> </ul> <p>Select a GPU from the list. The model instance will attempt to deploy to this GPU if resources permit.</p>"},{"location":"user-guide/model-management/#backend","title":"Backend","text":"<p>The inference backend. Currently, GPUStack supports three backends: llama-box, vLLM and vox-box. GPUStack automatically selects the backend based on the model's configuration.</p> <p>For more details, please refer to the Inference Backends section.</p>"},{"location":"user-guide/model-management/#backend-version","title":"Backend Version","text":"<p>Specify a backend version, such as <code>v1.0.0</code>. The version format and availability depend on the selected backend. This option is useful for ensuring compatibility or taking advantage of features introduced in specific backend versions. Refer to the Pinned Backend Versions section for more information.</p>"},{"location":"user-guide/model-management/#backend-parameters","title":"Backend Parameters","text":"<p>Input the parameters for the backend you want to customize when running the model. The parameter should be in the format <code>--parameter=value</code>, <code>--bool-parameter</code> or as separate fields for <code>--parameter</code> and <code>value</code>. For example, use <code>--ctx-size=8192</code> for llama-box.</p> <p>For full list of supported parameters, please refer to the Inference Backends section.</p>"},{"location":"user-guide/model-management/#allow-cpu-offloading","title":"Allow CPU Offloading","text":"<p>Note</p> <p>Available for llama-box backend only.</p> <p>After enabling CPU offloading, GPUStack prioritizes loading as many layers as possible onto the GPU to optimize performance. If GPU resources are limited, some layers will be offloaded to the CPU, with full CPU inference used only when no GPU is available.</p>"},{"location":"user-guide/model-management/#allow-distributed-inference-across-workers","title":"Allow Distributed Inference Across Workers","text":"<p>Note</p> <p>Available for llama-box backend only.</p> <p>Enable distributed inference across multiple workers. The primary Model Instance will communicate with backend instances on one or more other workers, offloading computation tasks to them.</p>"},{"location":"user-guide/openai-compatible-apis/","title":"OpenAI Compatible APIs","text":"<p>GPUStack serves OpenAI-compatible APIs using the <code>/v1-openai</code> path. Most of the APIs also work under the <code>/v1</code> path as an alias, except for the <code>models</code> endpoint, which is reserved for GPUStack management APIs.</p>"},{"location":"user-guide/openai-compatible-apis/#supported-endpoints","title":"Supported Endpoints","text":"<p>The following API endpoints are supported:</p> <ul> <li> List Models</li> <li> Create Completion</li> <li> Create Chat Completion</li> <li> Create Embeddings</li> <li> Create Image</li> <li> Create Image Edit</li> <li> Create Speech</li> <li> Create Transcription</li> </ul>"},{"location":"user-guide/openai-compatible-apis/#usage","title":"Usage","text":"<p>The following are examples using the APIs in different languages:</p>"},{"location":"user-guide/openai-compatible-apis/#curl","title":"curl","text":"<pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -d '{\n    \"model\": \"llama3\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a helpful assistant.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n      }\n    ],\n    \"stream\": true\n  }'\n</code></pre>"},{"location":"user-guide/openai-compatible-apis/#openai-python-api-library","title":"OpenAI Python API library","text":"<pre><code>from openai import OpenAI\n\nclient = OpenAI(base_url=\"http://myserver/v1-openai\", api_key=\"myapikey\")\n\ncompletion = client.chat.completions.create(\n  model=\"llama3\",\n  messages=[\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"Hello!\"}\n  ]\n)\n\nprint(completion.choices[0].message)\n</code></pre>"},{"location":"user-guide/openai-compatible-apis/#openai-node-api-library","title":"OpenAI Node API library","text":"<pre><code>const OpenAI = require(\"openai\");\n\nconst openai = new OpenAI({\n  apiKey: \"myapikey\",\n  baseURL: \"http://myserver/v1-openai\",\n});\n\nasync function main() {\n  const params = {\n    model: \"llama3\",\n    messages: [\n      {\n        role: \"system\",\n        content: \"You are a helpful assistant.\",\n      },\n      {\n        role: \"user\",\n        content: \"Hello!\",\n      },\n    ],\n  };\n  const chatCompletion = await openai.chat.completions.create(params);\n  console.log(chatCompletion.choices[0].message);\n}\nmain();\n</code></pre>"},{"location":"user-guide/pinned-backend-versions/","title":"Pinned Backend Versions","text":"<p>Inference engines in the generative AI domain are evolving rapidly to enhance performance and unlock new capabilities. This constant evolution provides exciting opportunities but also presents challenges for maintaining model compatibility and deployment stability.</p> <p>GPUStack allows you to pin inference backend versions to specific releases, offering a balance between staying up-to-date with the latest advancements and ensuring a reliable runtime environment. This feature is particularly beneficial in the following scenarios:</p> <ul> <li>Leveraging the newest backend features without waiting for a GPUStack update.</li> <li>Locking in a specific backend version to maintain compatibility with existing models.</li> <li>Assigning different backend versions to models with varying requirements.</li> </ul> <p>By pinning backend versions, you gain full control over your inference environment, enabling both flexibility and predictability in deployment.</p>"},{"location":"user-guide/pinned-backend-versions/#automatic-installation-of-pinned-backend-versions","title":"Automatic Installation of Pinned Backend Versions","text":"<p>To simplify deployment, GPUStack supports the automatic installation of pinned backend versions when feasible. The process depends on the type of backend:</p> <ol> <li>Prebuilt Binaries    For backends like <code>llama-box</code>, GPUStack downloads the specified version using the same mechanism as in GPUStack bootstrapping.</li> </ol> <p>Tip</p> <p>You can customize the download source using the <code>--tools-download-base-url</code> configuration option.</p> <ol> <li>Python-based Backends    For backends like <code>vLLM</code> and <code>vox-box</code>, GPUStack uses <code>pipx</code> to install the specified version in an isolated Python environment.</li> </ol> <p>Tip</p> <ul> <li>Ensure that <code>pipx</code> is installed on the worker nodes.</li> <li>If <code>pipx</code> is not in the system PATH, specify its location with the <code>--pipx-path</code> configuration option.</li> </ul> <p>This automation reduces manual intervention, allowing you to focus on deploying and using your models.</p>"},{"location":"user-guide/pinned-backend-versions/#manual-installation-of-pinned-backend-versions","title":"Manual Installation of Pinned Backend Versions","text":"<p>When automatic installation is not feasible or preferred, GPUStack provides a straightforward way to manually install specific versions of inference backends. Follow these steps:</p> <ol> <li>Prepare the Executable    Install the backend executable or link it under the GPUStack bin directory. The default locations are:</li> </ol> <ul> <li>Linux/macOS: <code>/var/lib/gpustack/bin</code></li> <li>Windows: <code>$env:AppData\\gpustack\\bin</code></li> </ul> <p>Tip</p> <p>You can customize the bin directory using the <code>--bin-dir</code> configuration option.</p> <ol> <li>Name the Executable    Ensure the executable is named in the following format:</li> </ol> <ul> <li>Linux/macOS: <code>&lt;backend&gt;_&lt;version&gt;</code></li> <li>Windows: <code>&lt;backend&gt;_&lt;version&gt;.exe</code></li> </ul> <p>For example, the vLLM executable for version v0.6.4 should be named <code>vllm_v0.6.4</code> on Linux.</p> <p>By following these steps, you can maintain full control over the backend installation process, ensuring that the correct version is used for your deployment.</p>"},{"location":"user-guide/rerank-api/","title":"Rerank API","text":"<p>In the context of Retrieval-Augmented Generation (RAG), reranking refers to the process of selecting the most relevant information from retrieved documents or knowledge sources before presenting them to the user or utilizing them for answer generation.</p> <p>GPUStack serves Jina compatible Rerank API using the <code>/v1/rerank</code> path.</p> <p>Note</p> <p>The Rerank API is only available when using the llama-box inference backend.</p>"},{"location":"user-guide/rerank-api/#supported-models","title":"Supported Models","text":"<p>The following models are available for reranking:</p> <ul> <li>bce-reranker-base_v1</li> <li>jina-reranker-v1-turbo-en</li> <li>jina-reranker-v1-tiny-en</li> <li>bge-reranker-v2-m3</li> <li>gte-multilingual-reranker-base \ud83e\uddea</li> <li>jina-reranker-v2-base-multilingual \ud83e\uddea</li> </ul>"},{"location":"user-guide/rerank-api/#usage","title":"Usage","text":"<p>The following is an example using the Rerank API:</p> <pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1/rerank \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -d '{\n        \"model\": \"bge-reranker-v2-m3\",\n        \"query\": \"What is a panda?\",\n        \"top_n\": 3,\n        \"documents\": [\n            \"hi\",\n            \"it is a bear\",\n            \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n        ]\n    }' | jq\n</code></pre> <p>Example output:</p> <pre><code>{\n  \"model\": \"bge-reranker-v2-m3\",\n  \"object\": \"list\",\n  \"results\": [\n    {\n      \"document\": {\n        \"text\": \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n      },\n      \"index\": 2,\n      \"relevance_score\": 1.951932668685913\n    },\n    {\n      \"document\": {\n        \"text\": \"it is a bear\"\n      },\n      \"index\": 1,\n      \"relevance_score\": -3.7347371578216553\n    },\n    {\n      \"document\": {\n        \"text\": \"hi\"\n      },\n      \"index\": 0,\n      \"relevance_score\": -6.157620906829834\n    }\n  ],\n  \"usage\": {\n    \"prompt_tokens\": 69,\n    \"total_tokens\": 69\n  }\n}\n</code></pre>"},{"location":"user-guide/user-management/","title":"User Management","text":"<p>GPUStack supports users of two roles: <code>Admin</code> and <code>User</code>. Admins can monitor system status, manage models, users, and system settings. Users can manage their own API keys and use the completion API.</p>"},{"location":"user-guide/user-management/#default-admin","title":"Default Admin","text":"<p>On bootstrap, GPUStack creates a default admin user. The initial password for the default admin is stored in <code>&lt;data-dir&gt;/initial_admin_password</code>. In the default setup, it should be <code>/var/lib/gpustack/initial_admin_password</code>. You can customize the default admin password by setting the <code>--bootstrap-password</code> parameter when starting <code>gpustack</code>.</p>"},{"location":"user-guide/user-management/#create-user","title":"Create User","text":"<ol> <li>Navigate to the <code>Users</code> page.</li> <li>Click the <code>Create User</code> button.</li> <li>Fill in <code>Name</code>, <code>Full Name</code>, <code>Password</code>, and select <code>Role</code> for the user.</li> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"user-guide/user-management/#update-user","title":"Update User","text":"<ol> <li>Navigate to the <code>Users</code> page.</li> <li>Find the user you want to edit.</li> <li>Click the <code>Edit</code> button in the <code>Operations</code> column.</li> <li>Update the attributes as needed.</li> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"user-guide/user-management/#delete-user","title":"Delete User","text":"<ol> <li>Navigate to the <code>Users</code> page.</li> <li>Find the user you want to delete.</li> <li>Click the ellipsis button in the <code>Operations</code> column, then select <code>Delete</code>.</li> <li>Confirm the deletion.</li> </ol>"},{"location":"user-guide/playground/","title":"Playground","text":"<p>GPUStack offers a playground UI where users can test and experiment with the APIs. Refer to each subpage for detailed instructions and information.</p>"},{"location":"user-guide/playground/audio/","title":"Audio Playground","text":"<p>The Audio Playground is a dedicated space for testing and experimenting with GPUStack\u2019s text-to-speech (TTS) and speech-to-text (STT) APIs. It allows users to interactively convert text to audio and audio to text, customize parameters, and review code examples for seamless API integration.</p>"},{"location":"user-guide/playground/audio/#text-to-speech","title":"Text to Speech","text":"<p>Switch to the \"Text to Speech\" tab to test TTS models.</p>"},{"location":"user-guide/playground/audio/#text-input","title":"Text Input","text":"<p>Enter the text you want to convert, then click the <code>Submit</code> button to generate the corresponding speech.</p> <p></p>"},{"location":"user-guide/playground/audio/#clear-text","title":"Clear Text","text":"<p>Click the <code>Clear</code> button to reset the text input and remove the generated speech.</p>"},{"location":"user-guide/playground/audio/#select-model","title":"Select Model","text":"<p>Select an available TTS model in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/audio/#customize-parameters","title":"Customize Parameters","text":"<p>Customize the voice and format of the audio output.</p> <p>Tip</p> <p>Supported voices may vary between models.</p>"},{"location":"user-guide/playground/audio/#view-code","title":"View Code","text":"<p>After experimenting with input text and parameters, click the <code>View Code</code> button to see how to call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/audio/#speech-to-text","title":"Speech to Text","text":"<p>Switch to the \"Speech to Text\" tab to test STT models.</p>"},{"location":"user-guide/playground/audio/#provide-audio-file","title":"Provide Audio File","text":"<p>You can provide audio for transcription in two ways:</p> <ol> <li>Upload an audio file.</li> <li>Record audio online.</li> </ol> <p>Note</p> <p>If the online recording is not available, it could be due to one of the following reasons:</p> <ol> <li>For HTTPS or <code>http://localhost</code> access, microphone permissions must be enabled in your browser.</li> <li> <p>For access via <code>http://{host IP}</code>, the URL must be added to your browser's trusted list.</p> <p>Example:   In Chrome, navigate to <code>chrome://flags/</code>, add the GPUStack URL to \"Insecure origins treated as secure,\" and enable this option.</p> </li> </ol> <p></p> <p></p>"},{"location":"user-guide/playground/audio/#select-model_1","title":"Select Model","text":"<p>Select an available STT model in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/audio/#copy-text","title":"Copy Text","text":"<p>Copy the transcription results generated by the model.</p>"},{"location":"user-guide/playground/audio/#customize-parameters_1","title":"Customize Parameters","text":"<p>Select the appropriate language for your audio file to optimize transcription accuracy.</p>"},{"location":"user-guide/playground/audio/#view-code_1","title":"View Code","text":"<p>After experimenting with audio files and parameters, click the <code>View Code</code> button to see how to call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/chat/","title":"Chat Playground","text":"<p>Interact with the chat completions API. The following is an example screenshot:</p> <p></p>"},{"location":"user-guide/playground/chat/#prompts","title":"Prompts","text":"<p>You can adjust the prompt messages on the left side of the playground. There are three role types of prompt messages: system, user, and assistant.</p> <ul> <li>System: Typically a predefined instruction or guidance that sets the context, defines the behavior, or imposes specific constraints on how the model should generate its responses.</li> <li>User: The input or query provided by the user (the person interacting with the LLM).</li> <li>Assistant: The response generated by the LLM.</li> </ul>"},{"location":"user-guide/playground/chat/#edit-system-message","title":"Edit System Message","text":"<p>You can add and edit the system message at the top of the playground.</p>"},{"location":"user-guide/playground/chat/#edit-user-and-assistant-messages","title":"Edit User and Assistant Messages","text":"<p>To add a user or assistant message, click the <code>New Message</code> button.</p> <p>To remove a user or assistant message, click the minus button at the right corner of the message.</p> <p>To change the role of a message, click the <code>User</code> or <code>Assistant</code> text at the beginning of the message.</p>"},{"location":"user-guide/playground/chat/#upload-image","title":"Upload Image","text":"<p>You can add images to the prompt by clicking the <code>Upload Image</code> button.</p>"},{"location":"user-guide/playground/chat/#clear-prompts","title":"Clear Prompts","text":"<p>Click the <code>Clear</code> button to clear all the prompts.</p>"},{"location":"user-guide/playground/chat/#select-model","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-right corner of the playground. Please refer to Model Management to learn about how to manage models.</p>"},{"location":"user-guide/playground/chat/#customize-parameters","title":"Customize Parameters","text":"<p>You can customize completion parameters in the <code>Parameters</code> section.</p>"},{"location":"user-guide/playground/chat/#do-completion","title":"Do Completion","text":"<p>You can do a completion by clicking the <code>Submit</code> button.</p>"},{"location":"user-guide/playground/chat/#view-code","title":"View Code","text":"<p>Once you've done experimenting with the prompts and parameters, you can click the <code>View Code</code> button to check how you can call the API with the same input by code. Code examples in <code>curl</code>, <code>Python</code>, and <code>Node.js</code> are provided.</p>"},{"location":"user-guide/playground/chat/#compare-playground","title":"Compare Playground","text":"<p>You can compare multiple models in the playground. The following is an example screenshot:</p> <p></p>"},{"location":"user-guide/playground/chat/#comparision-mode","title":"Comparision Mode","text":"<p>You can choose the number of models to compare by clicking the comparison view buttons, including 2, 3, 4 and 6-model comparison.</p>"},{"location":"user-guide/playground/chat/#prompts_1","title":"Prompts","text":"<p>You can adjust the prompt messages similar to the chat playground.</p>"},{"location":"user-guide/playground/chat/#upload-image_1","title":"Upload Image","text":"<p>You can add images to the prompt by clicking the <code>Upload Image</code> button.</p>"},{"location":"user-guide/playground/chat/#clear-prompts_1","title":"Clear Prompts","text":"<p>Click the <code>Clear</code> button to clear all the prompts.</p>"},{"location":"user-guide/playground/chat/#select-model_1","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-left corner of each model panel.</p>"},{"location":"user-guide/playground/chat/#customize-parameters_1","title":"Customize Parameters","text":"<p>You can customize completion parameters by clicking the settings button of each model.</p>"},{"location":"user-guide/playground/embedding/","title":"Embedding Playground","text":"<p>The Embedding Playground lets you test the model\u2019s ability to convert text into embeddings. It allows you to experiment with multiple text inputs, visualize embeddings, and review code examples for API integration.</p>"},{"location":"user-guide/playground/embedding/#add-text","title":"Add Text","text":"<p>Add at least two text entries and click the <code>Submit</code> button to generate embeddings.</p>"},{"location":"user-guide/playground/embedding/#batch-input-text","title":"Batch Input Text","text":"<p>Enable <code>Batch Input Mode</code> to automatically split multi-line text into separate entries based on line breaks. This is useful for processing multiple text snippets in a single operation.</p>"},{"location":"user-guide/playground/embedding/#visualization","title":"Visualization","text":"<p>Visualize the embedding results using PCA (Principal Component Analysis) to reduce dimensions and display them on a 2D plot. Results can be viewed in two formats:</p> <ol> <li>Chart - Display PCA results visually.</li> <li>JSON - View raw embeddings in JSON format.</li> </ol> <p>In the chart, the distance between points represents the similarity between corresponding texts. Closer points indicate higher similarity.</p> <p></p>"},{"location":"user-guide/playground/embedding/#clear","title":"Clear","text":"<p>Click the <code>Clear</code> button to reset text entries and clear the output.</p>"},{"location":"user-guide/playground/embedding/#select-model","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/embedding/#view-code","title":"View Code","text":"<p>After experimenting with the text inputs, click the <code>View Code</code> button to see how you can call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/image/","title":"Image Playground","text":"<p>The Image Playground is a dedicated space for testing and experimenting with GPUStack\u2019s image generation APIs. It allows users to interactively explore the capabilities of different models, customize parameters, and review code examples for seamless API integration.</p>"},{"location":"user-guide/playground/image/#prompt","title":"Prompt","text":"<p>You can input or randomly generate a prompt, then click the Submit button to generate an image.</p> <p></p>"},{"location":"user-guide/playground/image/#clear-prompt","title":"Clear Prompt","text":"<p>Click the <code>Clear</code> button to reset the prompt and remove the generated image.</p>"},{"location":"user-guide/playground/image/#select-model","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/image/#customize-parameters","title":"Customize Parameters","text":"<p>You can customize the image generation parameters by switching between two API styles:</p> <ol> <li>OpenAI-compatible mode.</li> <li>Advanced mode.</li> </ol> <p></p>"},{"location":"user-guide/playground/image/#advanced-parameters","title":"Advanced Parameters","text":"Parameter Default Description <code>Counts</code> <code>1</code> Number of images to generate. <code>Size</code> <code>512x512</code> The size of the generated image in 'widthxheight' format. <code>Sampler</code> <code>euler_a</code> The sampler algorithm for image generation. Options include 'euler_a', 'euler', 'heun', 'dpm2', 'dpm++2s_a', 'dpm++2m', 'dpm++2mv2', 'ipndm', 'ipndm_v', and 'lcm'. <code>Schedule</code> <code>discrete</code> The noise scheduling method. <code>Sampler Steps</code> <code>10</code> The number of sampling steps to perform. Higher values may improve image quality at the cost of longer processing time. <code>CFG Scale</code> <code>4.5</code> The scale for classifier-free guidance. A higher value increases adherence to the prompt. <code>Negative Prompt</code> (empty) A negative prompt to specify what the image should avoid. <code>Seed</code> (empty) Random seed. <p>Note</p> <p>The maximum image size is restricted by the model's deployment settings. See the diagram below:</p> <p></p>"},{"location":"user-guide/playground/image/#view-code","title":"View Code","text":"<p>After experimenting with prompts and parameters, click the <code>View Code</code> button to see how to call the API with the same inputs. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/rerank/","title":"Rerank Playground","text":"<p>The Rerank Playground allows you to test reranker models that reorder multiple texts based on their relevance to a query. Experiment with various input texts, customize parameters, and review code examples for API integration.</p>"},{"location":"user-guide/playground/rerank/#add-text","title":"Add Text","text":"<p>Add multiple text entries to the document for reranking.</p>"},{"location":"user-guide/playground/rerank/#bach-input-text","title":"Bach Input Text","text":"<p>Enable <code>Batch Input Mode</code> to split multi-line text into separate entries based on line breaks. This is useful for processing multiple text snippets efficiently.</p>"},{"location":"user-guide/playground/rerank/#clear","title":"Clear","text":"<p>Click the <code>Clear</code> button to reset the document and query results.</p>"},{"location":"user-guide/playground/rerank/#query","title":"Query","text":"<p>Input a query and click the <code>Submit</code> button to get a ranked list of texts based on their relevance to the query.</p> <p></p>"},{"location":"user-guide/playground/rerank/#select-model","title":"Select Model","text":"<p>Select an available reranker model in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/rerank/#customize-parameters","title":"Customize Parameters","text":"<p>In the parameter section, set <code>Top N</code> to specify the number of matching texts to retrieve.</p>"},{"location":"user-guide/playground/rerank/#view-code","title":"View Code","text":"<p>After experimenting with the input text and query, click the <code>View Code</code> button to see how to call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"api-reference/","title":"API Reference","text":"<p>GPUStack provides a built-in Swagger UI. You can access it by navigating to <code>&lt;gpustack-server-url&gt;/docs</code> in your browser to view and interact with the APIs.</p> <p></p>"},{"location":"architecture/","title":"Architecture","text":"<p>The following diagram shows the architecture of GPUStack:</p> <p></p>"},{"location":"architecture/#server","title":"Server","text":"<p>The GPUStack server consists of the following components:</p> <ul> <li>API Server: Provides a RESTful interface for clients to interact with the system. It handles authentication and authorization.</li> <li>Scheduler: Responsible for assigning model instances to workers.</li> <li>Model Controller: Manages the rollout and scaling of model instances to match the desired model replicas.</li> <li>HTTP Proxy: Routes completion API requests to backend inference servers.</li> </ul>"},{"location":"architecture/#worker","title":"Worker","text":"<p>GPUStack workers are responsible for:</p> <ul> <li>Running inference servers for model instances assigned to the worker.</li> <li>Reporting status to the server.</li> </ul>"},{"location":"architecture/#sql-database","title":"SQL Database","text":"<p>The GPUStack server connects to a SQL database as the datastore. GPUStack uses SQLite by default, but you can configure it to use an external PostgreSQL as well.</p>"},{"location":"architecture/#inference-server","title":"Inference Server","text":"<p>Inference servers are the backends that performs the inference tasks. GPUStack supports llama-box, vLLM and vox-box as the inference server.</p>"},{"location":"architecture/#rpc-server","title":"RPC Server","text":"<p>The RPC server enables running llama-box backend on a remote host. The Inference Server communicates with one or several instances of RPC server, offloading computations to these remote hosts. This setup allows for distributed LLM inference across multiple workers, enabling the system to load larger models even when individual resources are limited.</p>"},{"location":"code-of-conduct/","title":"Contributor Code of Conduct","text":""},{"location":"code-of-conduct/#our-pledge","title":"Our Pledge","text":"<p>We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.</p> <p>We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.</p>"},{"location":"code-of-conduct/#our-standards","title":"Our Standards","text":"<p>Examples of behavior that contributes to a positive environment for our community include:</p> <ul> <li>Demonstrating empathy and kindness toward other people</li> <li>Being respectful of differing opinions, viewpoints, and experiences</li> <li>Giving and gracefully accepting constructive feedback</li> <li>Accepting responsibility and apologizing to those affected by our mistakes,   and learning from the experience</li> <li>Focusing on what is best not just for us as individuals, but for the overall   community</li> </ul> <p>Examples of unacceptable behavior include:</p> <ul> <li>The use of sexualized language or imagery, and sexual attention or advances of   any kind</li> <li>Trolling, insulting or derogatory comments, and personal or political attacks</li> <li>Public or private harassment</li> <li>Publishing others' private information, such as a physical or email address,   without their explicit permission</li> <li>Other conduct which could reasonably be considered inappropriate in a   professional setting</li> </ul>"},{"location":"code-of-conduct/#enforcement-responsibilities","title":"Enforcement Responsibilities","text":"<p>Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.</p> <p>Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.</p>"},{"location":"code-of-conduct/#scope","title":"Scope","text":"<p>This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.</p>"},{"location":"code-of-conduct/#enforcement","title":"Enforcement","text":"<p>Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at contact@gpustack.ai. All complaints will be reviewed and investigated promptly and fairly.</p> <p>All community leaders are obligated to respect the privacy and security of the reporter of any incident.</p>"},{"location":"code-of-conduct/#enforcement-guidelines","title":"Enforcement Guidelines","text":"<p>Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:</p>"},{"location":"code-of-conduct/#1-correction","title":"1. Correction","text":"<p>Community Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.</p> <p>Consequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.</p>"},{"location":"code-of-conduct/#2-warning","title":"2. Warning","text":"<p>Community Impact: A violation through a single incident or series of actions.</p> <p>Consequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.</p>"},{"location":"code-of-conduct/#3-temporary-ban","title":"3. Temporary Ban","text":"<p>Community Impact: A serious violation of community standards, including sustained inappropriate behavior.</p> <p>Consequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.</p>"},{"location":"code-of-conduct/#4-permanent-ban","title":"4. Permanent Ban","text":"<p>Community Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.</p> <p>Consequence: A permanent ban from any sort of public interaction within the community.</p>"},{"location":"code-of-conduct/#attribution","title":"Attribution","text":"<p>This Code of Conduct is adapted from the Contributor Covenant, version 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.</p>"},{"location":"contributing/","title":"Contributing to GPUStack","text":"<p>Thanks for taking the time to contribute to GPUStack!</p> <p>Please review and follow the Code of Conduct.</p>"},{"location":"contributing/#filing-issues","title":"Filing Issues","text":"<p>If you find any bugs or are having any trouble, please search the reported issue as someone may have experienced the same issue, or we are actively working on a solution.</p> <p>If you can't find anything related to your issue, contact us by filing an issue. To help us diagnose and resolve, please include as much information as possible, including:</p> <ul> <li>Software: GPUStack version, installation method, operating system info, etc.</li> <li>Hardware: Node info, GPU info, etc.</li> <li>Steps to reproduce: Provide as much detail on how you got into the reported situation.</li> <li>Logs: Please include any relevant logs, such as server logs, worker logs, etc.</li> </ul>"},{"location":"contributing/#contributing-code","title":"Contributing Code","text":"<p>For setting up development environment, please refer to Development Guide.</p> <p>If you're fixing a small issue, you can simply submit a PR. However, if you're planning to submit a bigger PR to implement a new feature or fix a relatively complex bug, please open an issue that explains the change and the motivation for it. If you're addressing a bug, please explain how to reproduce it.</p>"},{"location":"contributing/#updating-documentation","title":"Updating Documentation","text":"<p>If you have any updates to our documentation, feel free to file an issue with the <code>documentation</code> label or make a pull request.</p>"},{"location":"development/","title":"Development Guide","text":""},{"location":"development/#prerequisites","title":"Prerequisites","text":"<p>Install Python (version 3.10 to 3.12).</p>"},{"location":"development/#set-up-environment","title":"Set Up Environment","text":"<pre><code>make install\n</code></pre>"},{"location":"development/#run","title":"Run","text":"<pre><code>poetry run gpustack\n</code></pre>"},{"location":"development/#build","title":"Build","text":"<pre><code>make build\n</code></pre> <p>And check artifacts in <code>dist</code>.</p>"},{"location":"development/#test","title":"Test","text":"<pre><code>make test\n</code></pre>"},{"location":"development/#update-dependencies","title":"Update Dependencies","text":"<pre><code>poetry add &lt;something&gt;\n</code></pre> <p>Or</p> <pre><code>poetry add --group dev &lt;something&gt;\n</code></pre> <p>For dev/testing dependencies.</p>"},{"location":"overview/","title":"GPUStack","text":"<p>GPUStack is an open-source GPU cluster manager for running AI models.</p>"},{"location":"overview/#key-features","title":"Key Features","text":"<ul> <li>Broad Hardware Compatibility: Run with different brands of GPUs in Apple MacBooks, Windows PCs, and Linux servers.</li> <li>Broad Model Support: From LLMs to diffusion models, audio, embedding, and reranker models.</li> <li>Scales with Your GPU Inventory: Easily add more GPUs or nodes to scale up your operations.</li> <li>Distributed Inference: Supports both single-node multi-GPU and multi-node inference and serving.</li> <li>Multiple Inference Backends: Supports llama-box (llama.cpp &amp; stable-diffusion.cpp), vox-box and vLLM as the inference backend.</li> <li>Lightweight Python Package: Minimal dependencies and operational overhead.</li> <li>OpenAI-compatible APIs: Serve APIs that are compatible with OpenAI standards.</li> <li>User and API key management: Simplified management of users and API keys.</li> <li>GPU metrics monitoring: Monitor GPU performance and utilization in real-time.</li> <li>Token usage and rate metrics: Track token usage and manage rate limits effectively.</li> </ul>"},{"location":"overview/#supported-platforms","title":"Supported Platforms","text":"<ul> <li> macOS</li> <li> Windows</li> <li> Linux</li> </ul>"},{"location":"overview/#supported-accelerators","title":"Supported Accelerators","text":"<ul> <li> Apple Metal (M-series chips)</li> <li> NVIDIA CUDA (Compute Capability 6.0 and above)</li> <li> Ascend CANN</li> <li> Moore Threads MUSA</li> </ul> <p>We plan to support the following accelerators in future releases.</p> <ul> <li> AMD ROCm</li> <li> Intel oneAPI</li> <li> Qualcomm AI Engine</li> </ul>"},{"location":"overview/#supported-models","title":"Supported Models","text":"<p>GPUStack uses llama-box (bundled llama.cpp and stable-diffusion.cpp server), vLLM and vox-box as the backends and supports a wide range of models. Models from the following sources are supported:</p> <ol> <li> <p>Hugging Face</p> </li> <li> <p>ModelScope</p> </li> <li> <p>Ollama Library</p> </li> <li> <p>Local File Path</p> </li> </ol>"},{"location":"overview/#example-models","title":"Example Models:","text":"Category Models Large Language Models(LLMs) Qwen, LLaMA, Mistral, Deepseek, Phi, Yi Vision Language Models(VLMs) Llama3.2-Vision, Pixtral , Qwen2-VL, LLaVA, InternVL2 Diffusion Models Stable Diffusion, FLUX Rerankers GTE, BCE, BGE, Jina Audio Models Whisper (speech-to-text), CosyVoice (text-to-speech) <p>For full list of supported models, please refer to the supported models section in the inference backends documentation.</p>"},{"location":"overview/#openai-compatible-apis","title":"OpenAI-Compatible APIs","text":"<p>GPUStack serves OpenAI compatible APIs. For details, please refer to OpenAI Compatible APIs</p>"},{"location":"quickstart/","title":"Quickstart","text":""},{"location":"quickstart/#installation","title":"Installation","text":""},{"location":"quickstart/#linux-or-macos","title":"Linux or macOS","text":"<p>GPUStack provides a script to install it as a service on systemd or launchd based systems. To install GPUStack using this method, just run:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s -\n</code></pre>"},{"location":"quickstart/#windows","title":"Windows","text":"<p>Run PowerShell as administrator (avoid using PowerShell ISE), then run the following command to install GPUStack:</p> <pre><code>Invoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre>"},{"location":"quickstart/#other-installation-methods","title":"Other Installation Methods","text":"<p>For manual installation, docker installation or detailed configuration options, please refer to the Installation Documentation.</p>"},{"location":"quickstart/#getting-started","title":"Getting Started","text":"<ol> <li>Run and chat with the llama3.2 model:</li> </ol> <pre><code>gpustack chat llama3.2 \"tell me a joke.\"\n</code></pre> <ol> <li>Run and generate an image with the stable-diffusion-v3-5-large-turbo model:</li> </ol> <p>Tip</p> <p>This command downloads the model (~12GB) from Hugging Face. The download time depends on your network speed. Ensure you have enough disk space and VRAM (12GB) to run the model. If you encounter issues, you can skip this step and move to the next one.</p> <pre><code>gpustack draw hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF:stable-diffusion-v3-5-large-turbo-Q4_0.gguf \\\n\"A minion holding a sign that says 'GPUStack'. The background is filled with futuristic elements like neon lights, circuit boards, and holographic displays. The minion is wearing a tech-themed outfit, possibly with LED lights or digital patterns. The sign itself has a sleek, modern design with glowing edges. The overall atmosphere is high-tech and vibrant, with a mix of dark and neon colors.\" \\\n--sample-steps 5 --show\n</code></pre> <p>Once the command completes, the generated image will appear in the default viewer. You can experiment with the prompt and CLI options to customize the output.</p> <p></p> <ol> <li>Open <code>http://myserver</code> in the browser to access the GPUStack UI. Log in to GPUStack with username <code>admin</code> and the default password. You can run the following command to get the password for the default setup:</li> </ol> <p>Linux or macOS</p> <pre><code>cat /var/lib/gpustack/initial_admin_password\n</code></pre> <p>Windows</p> <pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\initial_admin_password\" -Raw\n</code></pre> <ol> <li>Click <code>Playground</code> in the navigation menu. Now you can chat with the LLM in the UI playground.</li> </ol> <p></p> <ol> <li> <p>Click <code>API Keys</code> in the navigation menu, then click the <code>New API Key</code> button.</p> </li> <li> <p>Fill in the <code>Name</code> and click the <code>Save</code> button.</p> </li> <li> <p>Copy the generated API key and save it somewhere safe. Please note that you can only see it once on creation.</p> </li> <li> <p>Now you can use the API key to access the OpenAI-compatible API. For example, use curl as the following:</p> </li> </ol> <pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -d '{\n    \"model\": \"llama3.2\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a helpful assistant.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n      }\n    ],\n    \"stream\": true\n  }'\n</code></pre>"},{"location":"quickstart/#cleanup","title":"Cleanup","text":"<p>After you complete using the deployed models, you can go to the <code>Models</code> page in the GPUStack UI and delete the models to free up resources.</p>"},{"location":"scheduler/","title":"Scheduler","text":""},{"location":"scheduler/#summary","title":"Summary","text":"<p>The scheduler's primary responsibility is to calculate the resources required by models instance and to evaluate and select the optimal workers/GPUs for model instances through a series of strategies. This ensures that model instances can run efficiently. This document provides a detailed overview of the policies and processes used by the scheduler.</p>"},{"location":"scheduler/#scheduling-process","title":"Scheduling Process","text":""},{"location":"scheduler/#filtering-phase","title":"Filtering Phase","text":"<p>The filtering phase aims to narrow down the available workers or GPUs to those that meet specific criteria. The main policies involved are:</p> <ul> <li>Label Matching Policy</li> <li>Status Policy</li> <li>Resource Fit Policy</li> </ul>"},{"location":"scheduler/#label-matching-policy","title":"Label Matching Policy","text":"<p>This policy filters workers based on the label selectors configured for the model. If no label selectors are defined for the model, all workers are considered. Otherwise, the system checks whether the labels of each worker node match the model's label selectors, retaining only those workers that match.</p>"},{"location":"scheduler/#status-policy","title":"Status Policy","text":"<p>This policy filters workers based on their status, retaining only those that are in a READY state.</p>"},{"location":"scheduler/#resource-fit-policy","title":"Resource Fit Policy","text":"<p>The Resource Fit Policy is a critical strategy in the scheduling system, used to filter workers or GPUs based on resource compatibility. The goal of this policy is to ensure that model instances can run on the selected nodes without exceeding resource limits. The Resource Fit Policy prioritizes candidates in the following order:</p> <ul> <li>Single Worker Node, Single GPU Full Offload: Identifies candidates where a single GPU on a single worker can fully offload the model, which usually offers the best performance.</li> <li>Single Worker Node, Multiple GPU Full Offload: Identifies candidates where multiple GPUs on a single worker can fully the offload the model.</li> <li>Single Worker Node Partial Offload: Identifies candidates on a single worker that can handle a partial offload, used only when partial offloading is allowed.</li> <li>Distributed Inference Across Multiple Workers: Identifies candidates where a combination of GPUs across multiple workers can handle full or partial offloading, used only when distributed inference across nodes is permitted.</li> <li>Single Worker Node, CPU: When no GPUs are available, the system will use the CPU for inference, identifying candidates where memory resources on a single worker are sufficient.</li> </ul>"},{"location":"scheduler/#scoring-phase","title":"Scoring Phase","text":"<p>The scoring phase evaluates the filtered candidates, scoring them to select the optimal deployment location. The primary strategy involved is:</p> <ul> <li>Placement Strategy Policy</li> </ul>"},{"location":"scheduler/#placement-strategy-policy","title":"Placement Strategy Policy","text":"<ul> <li>Binpack</li> </ul> <p>This strategy aims to \"pack\" as many model instances as possible into the fewest number of \"bins\" (e.g., Workers/GPUs) to optimize resource utilization. The goal is to minimize the number of bins used while maximizing resource efficiency, ensuring each bin is filled as efficiently as possible without exceeding its capacity. Model instances are placed in the bin with the least remaining space to minimize leftover capacity in each bin.</p> <ul> <li>Spread</li> </ul> <p>This strategy seeks to distribute multiple model instances across different worker nodes as evenly as possible, improving system fault tolerance and load balancing.</p>"},{"location":"troubleshooting/","title":"Troubleshooting","text":""},{"location":"troubleshooting/#view-gpustack-logs","title":"View GPUStack Logs","text":"<p>If you installed GPUStack using the installation script, you can view GPUStack logs at the following path:</p>"},{"location":"troubleshooting/#linux-or-macos","title":"Linux or macOS","text":"<pre><code>/var/log/gpustack.log\n</code></pre>"},{"location":"troubleshooting/#windows","title":"Windows","text":"<pre><code>\"$env:APPDATA\\gpustack\\log\\gpustack.log\"\n</code></pre>"},{"location":"troubleshooting/#configure-log-level","title":"Configure Log Level","text":"<p>You can enable the DEBUG log level on <code>gpustack start</code> by setting the <code>--debug</code> parameter.</p> <p>You can configure log level of GPUStack server at runtime by running the following command on the server node:</p> <pre><code>curl -X PUT http://localhost/debug/log_level -d \"debug\"\n</code></pre>"},{"location":"troubleshooting/#reset-admin-password","title":"Reset Admin Password","text":"<p>In case you forgot the admin password, you can reset it by running the following command on the server node:</p> <pre><code>gpustack reset-admin-password\n</code></pre>"},{"location":"upgrade/","title":"Upgrade","text":"<p>You can upgrade GPUStack using the installation script or by manually installing the desired version of the GPUStack Python package.</p> <p>Note</p> <p>When upgrading, upgrade the GPUStack server first, then upgrade the workers.</p>"},{"location":"upgrade/#upgrade-gpustack-using-the-installation-script","title":"Upgrade GPUStack Using the Installation Script","text":"<p>To upgrade GPUStack from an older version, re-run the installation script using the same configuration options you originally used.</p> <p>Running the installation script will:</p> <ol> <li>Install the latest version of the GPUStack Python package.</li> <li>Update the system service (systemd, launchd, or Windows) init script to reflect the arguments passed to the installation script.</li> <li>Restart the GPUStack service.</li> </ol>"},{"location":"upgrade/#linux-and-macos","title":"Linux and macOS","text":"<p>For example, to upgrade GPUStack to the latest version on a Linux system and macOS:</p> <pre><code>curl -sfL https://get.gpustack.ai | &lt;EXISTING_INSTALL_ENV&gt; sh -s - &lt;EXISTING_GPUSTACK_ARGS&gt;\n</code></pre> <p>To upgrade to a specific version, specify the <code>INSTALL_PACKAGE_SPEC</code> environment variable similar to the <code>pip install</code> command:</p> <pre><code>curl -sfL https://get.gpustack.ai | INSTALL_PACKAGE_SPEC=gpustack==x.y.z &lt;EXISTING_INSTALL_ENV&gt; sh -s - &lt;EXISTING_GPUSTACK_ARGS&gt;\n</code></pre>"},{"location":"upgrade/#windows","title":"Windows","text":"<p>To upgrade GPUStack to the latest version on a Windows system:</p> <pre><code>$env:&lt;EXISTING_INSTALL_ENV&gt; = &lt;EXISTING_INSTALL_ENV_VALUE&gt;\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>To upgrade to a specific version:</p> <pre><code>$env:INSTALL_PACKAGE_SPEC = gpustack==x.y.z\n$env:&lt;EXISTING_INSTALL_ENV&gt; = &lt;EXISTING_INSTALL_ENV_VALUE&gt;\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } &lt;EXISTING_GPUSTACK_ARGS&gt;\"\n</code></pre>"},{"location":"upgrade/#docker-upgrade","title":"Docker Upgrade","text":"<p>If you installed GPUStack using Docker, upgrade to the a new version by pulling the Docker image with the desired version tag.</p> <p>For example:</p> <pre><code>docker pull gpustack/gpustack:vX.Y.Z\n</code></pre> <p>Then restart the GPUStack service with the new image.</p>"},{"location":"upgrade/#manual-upgrade","title":"Manual Upgrade","text":"<p>If you install GPUStack manually, upgrade using the common <code>pip</code> workflow.</p> <p>For example, to upgrade GPUStack to the latest version:</p> <pre><code>pip install --upgrade gpustack\n</code></pre> <p>Then restart the GPUStack service according to your setup.</p>"},{"location":"cli-reference/chat/","title":"gpustack chat","text":"<p>Chat with a large language model.</p> <pre><code>gpustack chat model [prompt]\n</code></pre>"},{"location":"cli-reference/chat/#positional-arguments","title":"Positional Arguments","text":"Name Description model The model to use for chat. prompt The prompt to send to the model. [Optional]"},{"location":"cli-reference/chat/#one-time-chat-with-a-prompt","title":"One-time Chat with a Prompt","text":"<p>If a prompt is provided, it performs a one-time inference. For example:</p> <pre><code>gpustack chat llama3 \"tell me a joke.\"\n</code></pre> <p>Example output:</p> <pre><code>Why couldn't the bicycle stand up by itself?\n\nBecause it was two-tired!\n</code></pre>"},{"location":"cli-reference/chat/#interactive-chat","title":"Interactive Chat","text":"<p>If the <code>prompt</code> argument is not provided, you can chat with the large language model interactively. For example:</p> <pre><code>gpustack chat llama3\n</code></pre> <p>Example output:</p> <pre><code>&gt;tell me a joke.\nHere's one:\n\nWhy couldn't the bicycle stand up by itself?\n\n(wait for it...)\n\nBecause it was two-tired!\n\nHope that made you smile!\n&gt;Do you have a better one?\nHere's another one:\n\nWhy did the scarecrow win an award?\n\n(think about it for a sec...)\n\nBecause he was outstanding in his field!\n\nHope that one stuck with you!\n\nDo you want to hear another one?\n&gt;\\quit\n</code></pre>"},{"location":"cli-reference/chat/#interactive-commands","title":"Interactive Commands","text":"<p>Followings are available commands in interactive chat:</p> <pre><code>Commands:\n  \\q or \\quit - Quit the chat\n  \\c or \\clear - Clear chat context in prompt\n  \\? or \\h or \\help - Print this help message\n</code></pre>"},{"location":"cli-reference/chat/#connect-to-external-gpustack-server","title":"Connect to External GPUStack Server","text":"<p>If you are not running <code>gpustack chat</code> on the server node, or if you are serving on a custom host or port, you should provide the following environment variables:</p> Name Description GPUSTACK_SERVER_URL URL of the GPUStack server, e.g., <code>http://myserver</code>. GPUSTACK_API_KEY GPUStack API key."},{"location":"cli-reference/download-tools/","title":"gpustack download-tools","text":"<p>Download dependency tools, including llama-box, gguf-parser, and fastfetch.</p> <pre><code>gpustack download-tools [OPTIONS]\n</code></pre>"},{"location":"cli-reference/download-tools/#configurations","title":"Configurations","text":"Flag Default Description <code>----tools-download-base-url</code> value (empty) Base URL to download dependency tools. <code>--save-archive</code> value (empty) Path to save downloaded tools as a tar archive. <code>--load-archive</code> value (empty) Path to load downloaded tools from a tar archive, instead of downloading. <code>--system</code> value Default is the current OS. Operating system to download tools for. Options: <code>linux</code>, <code>windows</code>, <code>macos</code>. <code>--arch</code> value Default is the current architecture. Architecture to download tools for. Options: <code>amd64</code>, <code>arm64</code>. <code>--device</code> value Default is the current device. Device to download tools for. Options: <code>cuda</code>, <code>mps</code>, <code>npu</code>, <code>musa</code>, <code>cpu</code>."},{"location":"cli-reference/draw/","title":"gpustack draw","text":"<p>Generate an image with a diffusion model.</p> <pre><code>gpustack draw [model] [prompt]\n</code></pre>"},{"location":"cli-reference/draw/#positional-arguments","title":"Positional Arguments","text":"Name Description model The model to use for image generation. prompt Text prompt to use for image generation. <p>The <code>model</code> can be either of the following:</p> <ol> <li>Name of a GPUStack model. You need to create a model in GPUStack before using it here.</li> <li>Reference to a Hugging Face GGUF diffusion model in Ollama style. When using this option, the model will be deployed if it is not already available. When not specified the default <code>Q4_0</code> tag is used. Examples:</li> </ol> <ul> <li><code>hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF</code></li> <li><code>hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF:FP16</code></li> <li><code>hf.co/gpustack/stable-diffusion-v3-5-large-turbo-GGUF:stable-diffusion-v3-5-large-turbo-Q4_0.gguf</code></li> </ul>"},{"location":"cli-reference/draw/#configurations","title":"Configurations","text":"Flag Default Description <code>--size</code> value <code>512x512</code> Size of the image to generate, specified as <code>widthxheight</code>. <code>--sampler</code> value <code>euler</code> Sampling method. Options include: euler_a, euler, heun, dpm2, dpm++2s_a, dpm++2m, lcm, etc. <code>--sample-steps</code> value (Empty) Number of sampling steps. <code>--cfg-scale</code> value (Empty) Classifier-free guidance scale for balancing prompt adherence and creativity. <code>--seed</code> value (Empty) Seed for random number generation. Useful for reproducibility. <code>--negative-prompt</code> value (Empty) Text prompt for what to avoid in the image. <code>--output</code> value (Empty) Path to save the generated image. <code>--show</code> <code>False</code> If True, opens the generated image in the default image viewer. <code>-d</code>, <code>--debug</code> <code>False</code> Enable debug mode."},{"location":"cli-reference/start/","title":"gpustack start","text":"<p>Run GPUStack server or worker.</p> <pre><code>gpustack start [OPTIONS]\n</code></pre>"},{"location":"cli-reference/start/#configurations","title":"Configurations","text":""},{"location":"cli-reference/start/#common-options","title":"Common Options","text":"Flag Default Description <code>--config-file</code> value (empty) Path to the YAML config file. <code>-d</code> value, <code>--debug</code> value <code>False</code> To enable debug mode, the short flag -d is not supported in Windows because this flag is reserved by PowerShell for CommonParameters. <code>--data-dir</code> value (empty) Directory to store data. Default is OS specific. <code>--cache-dir</code> value (empty) Directory to store cache (e.g., model files). Defaults to /cache. <code>-t</code> value, <code>--token</code> value Auto-generated. Shared secret used to add a worker. <code>--huggingface-token</code> value (empty) User Access Token to authenticate to the Hugging Face Hub. Can also be configured via the <code>HF_TOKEN</code> environment variable."},{"location":"cli-reference/start/#server-options","title":"Server Options","text":"Flag Default Description <code>--host</code> value <code>0.0.0.0</code> Host to bind the server to. <code>--port</code> value <code>80</code> Port to bind the server to. <code>--disable-worker</code> <code>False</code> Disable embedded worker. <code>--bootstrap-password</code> value Auto-generated. Initial password for the default admin user. <code>--database-url</code> value <code>sqlite:///&lt;data-dir&gt;/database.db</code> URL of the database. Example: postgresql://user:password@hostname:port/db_name <code>--ssl-keyfile</code> value (empty) Path to the SSL key file. <code>--ssl-certfile</code> value (empty) Path to the SSL certificate file. <code>--force-auth-localhost</code> <code>False</code> Force authentication for requests originating from localhost (127.0.0.1).When set to True, all requests from localhost will require authentication. <code>--ollama-library-base-url</code> <code>https://registry.ollama.ai</code> Base URL for the Ollama library. <code>--disable-update-check</code> <code>False</code> Disable update check."},{"location":"cli-reference/start/#worker-options","title":"Worker Options","text":"Flag Default Description <code>-s</code> value, <code>--server-url</code> value (empty) Server to connect to. <code>--worker-ip</code> value (empty) IP address of the worker node. Auto-detected by default. <code>--disable-metrics</code> <code>False</code> Disable metrics. <code>--disable-rpc-servers</code> <code>False</code> Disable RPC servers. <code>--metrics-port</code> value <code>10151</code> Port to expose metrics. <code>--worker-port</code> value <code>10150</code> Port to bind the worker to. Use a consistent value for all workers. <code>--log-dir</code> value (empty) Directory to store logs. <code>--system-reserved</code> value <code>\"{\\\"ram\\\": 2, \\\"vram\\\": 0}\"</code> The system reserves resources for the worker during scheduling, measured in GiB. By default, 2 GiB of RAM is reserved, Note: '{\\\"memory\\\": 2, \\\"gpu_memory\\\": 0}' is also supported, but it is deprecated and will be removed in future releases. <code>--tools-download-base-url</code> value Base URL for downloading dependency tools."},{"location":"cli-reference/start/#available-environment-variables","title":"Available Environment Variables","text":"<p>Most of the options can be set via environment variables. The environment variables are prefixed with <code>GPUSTACK_</code> and are in uppercase. For example, <code>--data-dir</code> can be set via the <code>GPUSTACK_DATA_DIR</code> environment variable.</p> <p>Below are additional environment variables that can be set:</p> Flag Description <code>HF_ENDPOINT</code> Hugging Face Hub endpoint. e.g., <code>https://hf-mirror.com</code>"},{"location":"cli-reference/start/#config-file","title":"Config File","text":"<p>You can configure start options using a YAML-format config file when starting GPUStack server or worker. Here is a complete example:</p> <pre><code># Common Options\ndebug: false\ndata_dir: /path/to/data_dir\ncache_dir: /path/to/cache_dir\ntoken: mytoken\n\n# Server Options\nhost: 0.0.0.0\nport: 80\ndisable_worker: false\ndatabase_url: postgresql://user:password@hostname:port/db_name\nssl_keyfile: /path/to/keyfile\nssl_certfile: /path/to/certfile\nforce_auth_localhost: false\nbootstrap_password: myadminpassword\nollama_library_base_url: https://registry.mycompany.com\ndisable_update_check: false\n\n# Worker Options\nserver_url: http://myserver\nworker_ip: 192.168.1.101\ndisable_metrics: false\ndisable_rpc_servers: false\nmetrics_port: 10151\nworker_port: 10150\nlog_dir: /path/to/log_dir\nsystem_reserved:\n  ram: 2\n  vram: 0\ntools_download_base_url: https://mirror.mycompany.com\n</code></pre>"},{"location":"installation/air-gapped-installation/","title":"Air-Gapped Installation","text":"<p>You can install GPUStack in an air-gapped environment. An air-gapped environment refers to a setup where GPUStack will be installed offline, behind a firewall, or behind a proxy.</p> <p>The following methods are available for installing GPUStack in an air-gapped environment:</p> <ul> <li>Docker Installation</li> <li>Manual Installation</li> </ul>"},{"location":"installation/air-gapped-installation/#docker-installation","title":"Docker Installation","text":"<p>When running GPUStack with Docker, it works out of the box in an air-gapped environment as long as the Docker images are available. To do this, follow these steps:</p> <ol> <li>Pull GPUStack Docker images in an online environment.</li> <li>Publish Docker images to a private registry.</li> <li>Refer to the Docker Installation guide to run GPUStack using Docker.</li> </ol>"},{"location":"installation/air-gapped-installation/#manual-installation","title":"Manual Installation","text":"<p>For manual installation, you need to prepare the required packages and tools in an online environment and then transfer them to the air-gapped environment.</p>"},{"location":"installation/air-gapped-installation/#prerequisites","title":"Prerequisites","text":"<p>Set up an online environment identical to the air-gapped environment, including OS, architecture, and Python version.</p>"},{"location":"installation/air-gapped-installation/#step-1-download-the-required-packages","title":"Step 1: Download the Required Packages","text":"<p>Run the following commands in an online environment:</p> <pre><code># On Windows (PowerShell):\n# $PACKAGE_SPEC = \"gpustack\"\n\n# Optional: To include extra dependencies (vllm, audio, all) or install a specific version\n# PACKAGE_SPEC=\"gpustack[all]\"\n# PACKAGE_SPEC=\"gpustack==0.4.0\"\nPACKAGE_SPEC=\"gpustack\"\n\n# Download all required packages\npip wheel $PACKAGE_SPEC -w gpustack_offline_packages\n\n# Install GPUStack to access its CLI\npip install gpustack\n\n# Download dependency tools and save them as an archive\ngpustack download-tools --save-archive gpustack_offline_tools.tar.gz\n</code></pre> <p>Optional: Additional Dependencies for macOS.</p> <pre><code># Deploying the speech-to-text CosyVoice model on macOS requires additional dependencies.\nbrew install openfst\nCPLUS_INCLUDE_PATH=$(brew --prefix openfst)/include\nLIBRARY_PATH=$(brew --prefix openfst)/lib\n\nAUDIO_DEPENDENCY_PACKAGE_SPEC=\"wetextprocessing\"\npip wheel $AUDIO_DEPENDENCY_PACKAGE_SPEC -w gpustack_audio_dependency_offline_packages\nmv gpustack_audio_dependency_offline_packages/* gpustack_offline_packages/ &amp;&amp; rm -rf gpustack_audio_dependency_offline_packages\n</code></pre> <p>Note</p> <p>This instruction assumes that the online environment uses the same GPU type as the air-gapped environment. If the GPU types differ, use the <code>--device</code> flag to specify the device type for the air-gapped environment. Refer to the download-tools command for more information.</p>"},{"location":"installation/air-gapped-installation/#step-2-transfer-the-packages","title":"Step 2: Transfer the Packages","text":"<p>Transfer the following files from the online environment to the air-gapped environment.</p> <ul> <li><code>gpustack_offline_packages</code> directory.</li> <li><code>gpustack_offline_tools.tar.gz</code> file.</li> </ul>"},{"location":"installation/air-gapped-installation/#step-3-install-gpustack","title":"Step 3: Install GPUStack","text":"<p>In the air-gapped environment, run the following commands:</p> <pre><code># Install GPUStack from the downloaded packages\npip install --no-index --find-links=gpustack_offline_packages gpustack\n\n# Load and apply the pre-downloaded tools archive\ngpustack download-tools --load-archive gpustack_offline_tools.tar.gz\n</code></pre> <p>Optional: Additional Dependencies for macOS.</p> <pre><code># Install the additional dependencies for speech-to-text CosyVoice model on macOS.\nbrew install openfst\n\npip install --no-index --find-links=gpustack_offline_packages wetextprocessing\n</code></pre> <p>Now you can run GPUStack by following the instructions in the Manual Installation guide.</p>"},{"location":"installation/docker-installation/","title":"Docker Installation","text":"<p>You can use the official Docker image to run GPUStack in a container. Installation using docker is supported on:</p> <ul> <li>Linux with Nvidia GPUs</li> </ul>"},{"location":"installation/docker-installation/#prerequisites","title":"Prerequisites","text":"<ul> <li>Docker</li> <li>Nvidia Container Toolkit</li> </ul>"},{"location":"installation/docker-installation/#run-gpustack-with-docker","title":"Run GPUStack with Docker","text":"<p>Run the following command to start the GPUStack server:</p> <pre><code>docker run -d --gpus all -p 80:80 --ipc=host \\\n    -v gpustack-data:/var/lib/gpustack gpustack/gpustack\n</code></pre> <p>Note</p> <p>You can either use the <code>--ipc=host</code> flag or <code>--shm-size</code> flag to allow the container to access the host\u2019s shared memory. It is used by vLLM and pyTorch to share data between processes under the hood, particularly for tensor parallel inference.</p> <p>You can set additional flags for the <code>gpustack start</code> command by appending them to the docker run command.</p> <p>For example, to start a GPUStack worker:</p> <pre><code>docker run -d --gpus all --ipc=host --network=host \\\n    gpustack/gpustack --server-url http://myserver --token mytoken\n</code></pre> <p>Note</p> <p>The <code>--network=host</code> flag is used to ensure that server is accessible to the worker and inference services running on it. Alternatively, you can set <code>--worker-ip &lt;host-ip&gt; -p 10150:10150 -p 40000-41024:40000-41024</code> to expose relevant ports.</p> <p>For configuration details, please refer to the CLI Reference.</p>"},{"location":"installation/docker-installation/#run-gpustack-with-docker-compose","title":"Run GPUStack with Docker Compose","text":"<p>Get the docker-compose file from GPUStack repository, run the following command to start the GPUStack server:</p> <pre><code>docker-compose up -d\n</code></pre> <p>You can update the <code>docker-compose.yml</code> file to customize the command while starting a GPUStack worker.</p>"},{"location":"installation/docker-installation/#build-your-own-docker-image","title":"Build Your Own Docker Image","text":"<p>The official Docker image is built with CUDA 12.4. If you want to use a different version of CUDA, you can build your own Docker image.</p> <pre><code># Example Dockerfile\nARG CUDA_VERSION=12.4.1\n\nFROM nvidia/cuda:$CUDA_VERSION-cudnn-runtime-ubuntu22.04\n\nENV DEBIAN_FRONTEND=noninteractive\n\nRUN apt-get update &amp;&amp; apt-get install -y \\\n    wget \\\n    tzdata \\\n    python3 \\\n    python3-pip \\\n    &amp;&amp; rm -rf /var/lib/apt/lists/*\n\n\nRUN pip3 install gpustack[all] &amp;&amp; \\\n    pip3 cache purge\n\nENTRYPOINT [ \"gpustack\", \"start\" ]\n</code></pre> <p>Run the following command to build the Docker image:</p> <pre><code>docker build -t my/gpustack --build-arg CUDA_VERSION=12.0.0 .\n</code></pre>"},{"location":"installation/installation-requirements/","title":"Installation Requirements","text":"<p>This page describes the software and networking requirements for the nodes where GPUStack will be installed.</p>"},{"location":"installation/installation-requirements/#python-requirements","title":"Python Requirements","text":"<p>GPUStack requires Python version 3.10 to 3.12.</p>"},{"location":"installation/installation-requirements/#operating-system-requirements","title":"Operating System Requirements","text":"<p>GPUStack is supported on the following operating systems:</p> <ul> <li> macOS</li> <li> Windows</li> <li> Linux</li> </ul> <p>GPUStack has been tested and verified to work on the following operating systems:</p> OS Versions Windows 10, 11 Ubuntu &gt;= 20.04 Debian &gt;= 11 RHEL &gt;= 8 Rocky &gt;= 8 Fedora &gt;= 36 OpenSUSE &gt;= 15.3 (leap) OpenEuler &gt;= 22.03 <p>Note</p> <p>The installation of GPUStack worker on a Linux system requires that the GLIBC version be 2.29 or higher. If your system uses a lower GLIBC version, consider using the Docker Installation method as an alternative.</p>"},{"location":"installation/installation-requirements/#supported-architectures","title":"Supported Architectures","text":"<p>GPUStack supports both AMD64 and ARM64 architectures, with the following notes:</p> <ul> <li>On Linux and macOS, when using Python versions below 3.12, ensure that the installed Python distribution corresponds to your system architecture.</li> <li>On Windows, please use the AMD64 distribution of Python, as wheel packages for certain dependencies are unavailable for ARM64. If you use tools like <code>conda</code>, this will be handled automatically, as conda installs the AMD64 distribution by default.</li> </ul>"},{"location":"installation/installation-requirements/#accelerator-runtime-requirements","title":"Accelerator Runtime Requirements","text":"<p>GPUStack supports the following accelerators:</p> <ul> <li> Apple Metal (M-series chips)</li> <li> NVIDIA CUDA (Compute Capability 6.0 and above)</li> <li> Ascend CANN</li> <li> Moore Threads MUSA</li> </ul> <p>Ensure all necessary drivers and libraries are installed on the system prior to installing GPUStack.</p>"},{"location":"installation/installation-requirements/#nvidia-cuda","title":"NVIDIA CUDA","text":"<p>To use NVIDIA CUDA as an accelerator, ensure the following components are installed:</p> <ul> <li>NVIDIA CUDA Toolkit</li> <li>NVIDIA cuBLAS (Optional, required for audio models)</li> <li>NVIDIA cuDNN (Optional, required for audio models)</li> <li>NVIDIA Container Toolkit (Optional, required for docker installation)</li> </ul>"},{"location":"installation/installation-requirements/#ascend-cann","title":"Ascend CANN","text":"<p>For Ascend CANN as an accelerator, ensure the following components are installed:</p> <ul> <li>Ascend NPU driver &amp; firmware</li> <li>Ascend CANN Toolkit &amp; kernels</li> </ul>"},{"location":"installation/installation-requirements/#musa","title":"MUSA","text":"<p>To use Moore Threads MUSA as an accelerator, ensure the following components are installed:</p> <ul> <li>MUSA SDK</li> <li>MT Container Toolkits(Optional, required for docker installation)</li> </ul>"},{"location":"installation/installation-requirements/#networking-requirements","title":"Networking Requirements","text":""},{"location":"installation/installation-requirements/#connectivity-requirements","title":"Connectivity Requirements","text":"<p>The following network connectivity is required to ensure GPUStack functions properly:</p> <p>Server-to-Worker: The server must be able to reach the workers for proxying inference requests.</p> <p>Worker-to-Server: Workers must be able to reach the server to register themselves and send updates.</p> <p>Worker-to-Worker: Necessary for distributed inference across multiple workers</p>"},{"location":"installation/installation-requirements/#port-requirements","title":"Port Requirements","text":"<p>GPUStack uses the following ports for communication:</p> <p>Server Ports</p> Port Description TCP 80 Default port for the GPUStack UI and API endpoints TCP 443 Default port for the GPUStack UI and API endpoints (when TLS is enabled) <p>Worker Ports</p> Port Description TCP 10150 Default port for the GPUStack worker TCP 10151 Default port for exposing metrics TCP 40000-41024 Port range allocated for inference services"},{"location":"installation/installation-script/","title":"Installation Script","text":""},{"location":"installation/installation-script/#linux-and-macos","title":"Linux and macOS","text":"<p>You can use the installation script available at <code>https://get.gpustack.ai</code> to install GPUStack as a service on systemd and launchd based systems.</p> <p>You can set additional environment variables and CLI flags when running the script. The following are examples running the installation script with different configurations:</p> <pre><code># Run server.\ncurl -sfL https://get.gpustack.ai | sh -s -\n\n# Run server without the embedded worker.\ncurl -sfL https://get.gpustack.ai | sh -s - --disable-worker\n\n# Run server with TLS.\ncurl -sfL https://get.gpustack.ai | sh -s - --ssl-keyfile /path/to/keyfile --ssl-certfile /path/to/certfile\n\n# Run server with external postgresql database.\ncurl -sfL https://get.gpustack.ai | sh -s - --database-url \"postgresql://username:password@host:port/database_name\"\n\n# Run worker with specified IP.\ncurl -sfL https://get.gpustack.ai | sh -s - --server-url http://myserver --token mytoken --worker-ip 192.168.1.100\n\n# Install with a custom index URL.\ncurl -sfL https://get.gpustack.ai | INSTALL_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple sh -s -\n\n# Install a custom wheel package other than releases form pypi.org.\ncurl -sfL https://get.gpustack.ai | INSTALL_PACKAGE_SPEC=https://repo.mycompany.com/my-gpustack.whl sh -s -\n\n# Install a specific version with extra audio dependencies.\ncurl -sfL https://get.gpustack.ai | INSTALL_PACKAGE_SPEC=gpustack[audio]==0.4.0 sh -s -\n</code></pre>"},{"location":"installation/installation-script/#windows","title":"Windows","text":"<p>You can use the installation script available at <code>https://get.gpustack.ai</code> to install GPUStack as a service on Windows Service Manager.</p> <p>You can set additional environment variables and CLI flags when running the script. The following are examples running the installation script with different configurations:</p> <pre><code># Run server.\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n\n# Run server without the embedded worker.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --disable-worker\"\n\n# Run server with TLS.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --ssl-keyfile 'C:\\path\\to\\keyfile' --ssl-certfile 'C:\\path\\to\\certfile'\"\n\n\n# Run server with external postgresql database.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --database-url 'postgresql://username:password@host:port/database_name'\"\n\n# Run worker with specified IP.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --server-url 'http://myserver' --token 'mytoken' --worker-ip '192.168.1.100'\"\n\n# Run worker with customize reserved resource.\nInvoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --server-url 'http://myserver' --token 'mytoken' --system-reserved '{\"\"ram\"\":5, \"\"vram\"\":5}'\"\n\n# Install with a custom index URL.\n$env:INSTALL_INDEX_URL = \"https://pypi.tuna.tsinghua.edu.cn/simple\"\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n\n# Install a custom wheel package other than releases form pypi.org.\n$env:INSTALL_PACKAGE_SPEC = \"https://repo.mycompany.com/my-gpustack.whl\"\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n\n# Install a specific version with extra audio dependencies.\n$env:INSTALL_PACKAGE_SPEC = \"gpustack[audio]==0.4.0\"\nInvoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>Warning</p> <p>Avoid using PowerShell ISE as it is not compatible with the installation script.</p>"},{"location":"installation/installation-script/#available-environment-variables-for-the-installation-script","title":"Available Environment Variables for the Installation Script","text":"Name Default Description <code>INSTALL_INDEX_URL</code> (empty) Base URL of the Python Package Index. <code>INSTALL_PACKAGE_SPEC</code> <code>gpustack[all]</code> or <code>gpustack[audio]</code> The package spec to install. The install script will automatically decide based on the platform. It supports PYPI package names, URLs, and local paths. See the pip install documentation for details. <ul><li><code>gpustack[all]</code>: With all inference backends: llama-box, vllm, vox-box.</li><li><code>gpustack[vllm]</code>: With inference backends: llama-box, vllm.</li><li><code>gpustack[audio]</code>: With inference backends: llama-box, vox-box.</li></ul> <code>INSTALL_SKIP_POST_CHECK</code> (empty) If set to 1, the installation script will skip the post-installation check."},{"location":"installation/installation-script/#set-environment-variables-for-the-gpustack-service","title":"Set Environment Variables for the GPUStack Service","text":"<p>You can set environment variables for the GPUStack service in an environment file located at:</p> <ul> <li>Linux and macOS: <code>/etc/default/gpustack</code></li> <li>Windows: <code>$env:APPDATA\\gpustack\\gpustack.env</code></li> </ul> <p>The following is an example of the content of the file:</p> <pre><code>HF_TOKEN=\"mytoken\"\nHF_ENDPOINT=\"https://my-hf-endpoint\"\n</code></pre> <p>Note</p> <p>Unlike Systemd, Launchd and Windows services do not natively support reading environment variables from a file. Configuration via the environment file is implemented by the installation script. It reads the file and applies the variables to the service configuration. After modifying the environment file on Windows and macOS, you need to re-run the installation script to apply changes to the GPUStack service.</p>"},{"location":"installation/installation-script/#available-cli-flags","title":"Available CLI Flags","text":"<p>The appended CLI flags of the installation script are passed directly as flags for the <code>gpustack start</code> command. You can refer to the CLI Reference for details.</p>"},{"location":"installation/installation-script/#install-server","title":"Install Server","text":"<p>To set up the GPUStack server (the management node), install GPUStack without the <code>--server-url</code> flag. By default, the GPUStack server includes an embedded worker. To disable this embedded worker on the server, use the <code>--disable-worker</code> flag.</p>"},{"location":"installation/installation-script/#install-worker","title":"Install Worker","text":"<p>To form a cluster, you can add GPUStack workers on additional nodes. Install GPUStack with the <code>--server-url</code> flag to specify the server' address and the <code>--token</code> flag for worker authenticate.</p> <p>Examples are as follows:</p>"},{"location":"installation/installation-script/#linux-or-macos","title":"Linux or macOS","text":"<pre><code>curl -sfL https://get.gpustack.ai | sh -s - --server-url http://myserver --token mytoken\n</code></pre> <p>In the default setup, you can run the following on the server node to get the token used for adding workers:</p> <pre><code>cat /var/lib/gpustack/token\n</code></pre>"},{"location":"installation/installation-script/#windows_1","title":"Windows","text":"<pre><code>Invoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } -- --server-url http://myserver --token mytoken\"\n</code></pre> <p>In the default setup, you can run the following on the server node to get the token used for adding workers:</p> <pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\token\" -Raw\n</code></pre>"},{"location":"installation/manual-installation/","title":"Manual Installation","text":""},{"location":"installation/manual-installation/#prerequites","title":"Prerequites:","text":"<p>Install Python version 3.10 to 3.12.</p>"},{"location":"installation/manual-installation/#install-gpustack-cli","title":"Install GPUStack CLI","text":"<p>Run the following to install GPUStack:</p> <pre><code># You can add extra dependencies, options are \"vllm\", \"audio\" and \"all\".\n# e.g., gpustack[all]\npip install gpustack\n</code></pre> <p>To verify, run:</p> <pre><code>gpustack version\n</code></pre>"},{"location":"installation/manual-installation/#run-gpustack","title":"Run GPUStack","text":"<p>Run the following command to start the GPUStack server:</p> <pre><code>gpustack start\n</code></pre> <p>By default, GPUStack uses <code>/var/lib/gpustack</code> as the data directory so you need <code>sudo</code> or proper permission for that. You can also set a custom data directory by running:</p> <pre><code>gpustack start --data-dir mypath\n</code></pre>"},{"location":"installation/manual-installation/#run-gpustack-as-a-system-service","title":"Run GPUStack as a System Service","text":"<p>A recommended way is to run GPUStack as a startup service. For example, using systemd:</p> <p>Create a service file in <code>/etc/systemd/system/gpustack.service</code>:</p> <pre><code>[Unit]\nDescription=GPUStack Service\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nEnvironmentFile=-/etc/default/%N\nExecStart=gpustack start\nRestart=always\nRestartSec=3\nStandardOutput=append:/var/log/gpustack.log\nStandardError=append:/var/log/gpustack.log\n\n[Install]\nWantedBy=multi-user.target\n</code></pre> <p>Then start GPUStack:</p> <pre><code>systemctl daemon-reload\nsystemctl enable gpustack\n</code></pre>"},{"location":"installation/uninstallation/","title":"Uninstallation","text":""},{"location":"installation/uninstallation/#uninstallation-script","title":"Uninstallation Script","text":"<p>Warning</p> <p>Uninstallation script deletes the data in local datastore(sqlite), configuration, model cache, and all of the scripts and CLI tools. It does not remove any data from external datastores.</p> <p>If you installed GPUStack using the installation script, a script to uninstall GPUStack was generated during installation.</p>"},{"location":"installation/uninstallation/#linux-or-macos","title":"Linux or macOS","text":"<p>Run the following command to uninstall GPUStack:</p> <pre><code>sudo /var/lib/gpustack/uninstall.sh\n</code></pre>"},{"location":"installation/uninstallation/#windows","title":"Windows","text":"<p>Run the following command in PowerShell to uninstall GPUStack:</p> <pre><code>Set-ExecutionPolicy Bypass -Scope Process -Force; &amp; \"$env:APPDATA\\gpustack\\uninstall.ps1\"\n</code></pre>"},{"location":"installation/uninstallation/#manual-uninstallation","title":"Manual Uninstallation","text":"<p>If you install GPUStack manually, the followings are example commands to uninstall GPUStack. You can modify according to your setup:</p> <pre><code># Stop and remove the service.\nsystemctl stop gpustack.service\nrm /etc/systemd/system/gpustack.service\nsystemctl daemon-reload\n# Uninstall the CLI.\npip uninstall gpustack\n# Remove the data directory.\nrm -rf /var/lib/gpustack\n</code></pre>"},{"location":"tutorials/creating-text-embeddings/","title":"Creating Text Embeddings","text":"<p>Text embeddings are numerical representations of text that capture semantic meaning, enabling machines to understand relationships and similarities between different pieces of text. In essence, they transform text into vectors in a continuous space, where texts with similar meanings are positioned closer together. Text embeddings are widely used in applications such as natural language processing, information retrieval, and recommendation systems.</p> <p>In this tutorial, we will demonstrate how to deploy embedding models in GPUStack and generate text embeddings using the deployed models.</p>"},{"location":"tutorials/creating-text-embeddings/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul>"},{"location":"tutorials/creating-text-embeddings/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Enable the <code>GGUF</code> checkbox to filter models by GGUF format.</li> <li>Use the search bar in the top left to search for the model name <code>CompendiumLabs/bge-small-en-v1.5-gguf</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/creating-text-embeddings/#step-2-generate-an-api-key","title":"Step 2: Generate an API Key","text":"<p>We will use the GPUStack API to generate text embeddings, and an API key is required:</p> <ol> <li>Navigate to the <code>API Keys</code> page in the GPUStack UI.</li> <li>Click the <code>New API Key</code> button.</li> <li>Enter a name for the API key and click the <code>Save</code> button.</li> <li>Copy the generated API key. You can only view the API key once, so make sure to save it securely.</li> </ol>"},{"location":"tutorials/creating-text-embeddings/#step-3-generate-text-embeddings","title":"Step 3: Generate Text Embeddings","text":"<p>With the model deployed and an API key, you can generate text embeddings via the GPUStack API. Here is an example script using <code>curl</code>:</p> <pre><code>export SERVER_URL=&lt;your-server-url&gt;\nexport GPUSTACK_API_KEY=&lt;your-api-key&gt;\ncurl $SERVER_URL/v1-openai/embeddings \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"input\": \"The food was delicious and the waiter...\",\n    \"model\": \"bge-small-en-v1.5\",\n    \"encoding_format\": \"float\"\n  }'\n</code></pre> <p>Replace <code>&lt;your-server-url&gt;</code> with the URL of your GPUStack server and <code>&lt;your-api-key&gt;</code> with the API key you generated in the previous step.</p> <p>Example response:</p> <pre><code>{\n  \"data\": [\n    {\n      \"embedding\": [\n        -0.012189436703920364, 0.016934078186750412, 0.003965042531490326,\n        -0.03453584015369415, -0.07623119652271271, -0.007116147316992283,\n        0.11278388649225235, 0.019714849069714546, 0.010370955802500248,\n        -0.04219457507133484, -0.029902394860982895, 0.01122555136680603,\n        0.022912170737981796, 0.031186765059828758, 0.006303929258137941,\n        # ... additional values\n      ],\n      \"index\": 0,\n      \"object\": \"embedding\"\n    }\n  ],\n  \"model\": \"bge-small-en-v1.5\",\n  \"object\": \"list\",\n  \"usage\": { \"prompt_tokens\": 12, \"total_tokens\": 12 }\n}\n</code></pre>"},{"location":"tutorials/inference-on-cpus/","title":"Inference on CPUs","text":"<p>GPUStack supports inference on CPUs, offering flexibility when GPU resources are limited or when model sizes exceed available GPU memory. The following CPU inference modes are available:</p> <ul> <li>CPU+GPU Hybrid Inference: Enables partial acceleration by offloading portions of large models to the CPU when VRAM capacity is insufficient.</li> <li>Full CPU Inference: Operates entirely on CPU when no GPU resources are available.</li> </ul> <p>Note</p> <p>CPU inference is supported when using the llama-box (llama.cpp) backend.</p> <p>To deploy a model with CPU offloading, enable the <code>Allow CPU Offloading</code> option in the deployment configuration (this setting is enabled by default).</p> <p></p> <p>After deployment, you can view the number of model layers offloaded to the CPU.</p> <p></p>"},{"location":"tutorials/inference-with-function-calling/","title":"Inference with Function Calling","text":"<p>Function calling allows you to connect models to external tools and systems. This is useful for many things such as empowering AI assistants with capabilities, or building deep integrations between your applications and the models.</p> <p>In this tutorial, you\u2019ll learn how to set up and use function calling within GPUStack to extend your AI\u2019s capabilities.</p> <p>Note</p> <ol> <li>Function calling is supported in the vLLM inference backend.</li> <li>Function calling is essentially achieved through prompt engineering, requiring models to be trained with internalized templates to enable this capability. Therefore, not all LLMs support function calling.</li> </ol>"},{"location":"tutorials/inference-with-function-calling/#prerequisites","title":"Prerequisites","text":"<p>Before proceeding, ensure the following:</p> <ul> <li>GPUStack is installed and running.</li> <li>A Linux worker node with a GPU is available. We'll use Qwen2.5-7B-Instruct as the model for this tutorial. The model requires a GPU with at least 18GB VRAM.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul>"},{"location":"tutorials/inference-with-function-calling/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI and click the <code>Deploy Model</code> button. In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar to find the <code>Qwen/Qwen2.5-7B-Instruct</code> model.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button and add the following parameters:</li> </ol> <ul> <li><code>--enable-auto-tool-choice</code></li> <li><code>--tool-call-parser=hermes</code></li> </ul> <ol> <li>Click the <code>Save</code> button to deploy the model.</li> </ol> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p>"},{"location":"tutorials/inference-with-function-calling/#step-2-generate-an-api-key","title":"Step 2: Generate an API Key","text":"<p>We will use the GPUStack API to interact with the model. To do this, you need to generate an API key:</p> <ol> <li>Navigate to the <code>API Keys</code> page in the GPUStack UI.</li> <li>Click the <code>New API Key</code> button.</li> <li>Enter a name for the API key and click the <code>Save</code> button.</li> <li>Copy the generated API key for later use.</li> </ol>"},{"location":"tutorials/inference-with-function-calling/#step-3-do-inference","title":"Step 3: Do Inference","text":"<p>With the model deployed and an API key, you can call the model via the GPUStack API. Here is an example script using <code>curl</code> (replace <code>&lt;your-server-url&gt;</code> with your GPUStack server URL and <code>&lt;your-api-key&gt;</code> with the API key generated in the previous step):</p> <pre><code>export GPUSTACK_SERVER_URL=&lt;your-server-url&gt;\nexport GPUSTACK_API_KEY=&lt;your-api-key&gt;\ncurl $GPUSTACK_SERVER_URL/v1-openai/chat/completions \\\n-H \"Content-Type: application/json\" \\\n-H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n-d '{\n  \"model\": \"qwen2.5-7b-instruct\",\n  \"messages\": [\n    {\n      \"role\": \"user\",\n      \"content\": \"What'\\''s the weather like in Boston today?\"\n    }\n  ],\n  \"tools\": [\n    {\n      \"type\": \"function\",\n      \"function\": {\n        \"name\": \"get_current_weather\",\n        \"description\": \"Get the current weather in a given location\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"location\": {\n              \"type\": \"string\",\n              \"description\": \"The city and state, e.g. San Francisco, CA\"\n            },\n            \"unit\": {\n              \"type\": \"string\",\n              \"enum\": [\"celsius\", \"fahrenheit\"]\n            }\n          },\n          \"required\": [\"location\"]\n        }\n      }\n    }\n  ],\n  \"tool_choice\": \"auto\"\n}'\n</code></pre> <p>Example response:</p> <pre><code>{\n  \"model\": \"qwen2.5-7b-instruct\",\n  \"choices\": [\n    {\n      \"index\": 0,\n      \"message\": {\n        \"role\": \"assistant\",\n        \"content\": null,\n        \"tool_calls\": [\n          {\n            \"id\": \"chatcmpl-tool-b99d32848b324eaea4bac5a5830d00b8\",\n            \"type\": \"function\",\n            \"function\": {\n              \"name\": \"get_current_weather\",\n              \"arguments\": \"{\\\"location\\\": \\\"Boston, MA\\\", \\\"unit\\\": \\\"fahrenheit\\\"}\"\n            }\n          }\n        ]\n      },\n      \"finish_reason\": \"tool_calls\"\n    }\n  ],\n  \"usage\": {\n    \"prompt_tokens\": 212,\n    \"total_tokens\": 242,\n    \"completion_tokens\": 30\n  }\n}\n</code></pre>"},{"location":"tutorials/performing-distributed-inference-across-workers/","title":"Performing Distributed Inference Across Workers","text":"<p>This tutorial will guide you through the process of configuring and running distributed inference across multiple workers using GPUStack. Distributed inference allows you to handle larger language models by distributing the computational workload among multiple workers. This is particularly useful when individual workers do not have sufficient resources, such as VRAM, to run the entire model independently.</p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#prerequisites","title":"Prerequisites","text":"<p>Before proceeding, ensure the following:</p> <ul> <li>GPUStack is installed and running. Refer to the Setting Up a Multi-node GPUStack Cluster tutorial if needed.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul> <p>In this tutorial, we\u2019ll assume a cluster with two nodes, each equipped with an NVIDIA P40 GPU (22GB VRAM), as shown in the following image:</p> <p></p> <p>We aim to run a large language model that requires more VRAM than a single worker can provide. For this tutorial, we\u2019ll use the <code>Qwen/Qwen2.5-72B-Instruct</code> model with the <code>q2_k</code> quantization format. The required resources for running this model can be estimated using the gguf-parser tool:</p> <pre><code>$ gguf-parser --hf-repo Qwen/Qwen2.5-72B-Instruct-GGUF --hf-file qwen2.5-72b-instruct-q2_k-00001-of-00007.gguf --ctx-size=8192 --in-short --skip-architecture --skip-metadata --skip-tokenizer\n\n+--------------------------------------------------------------------------------------+\n| ESTIMATE                                                                             |\n+----------------------------------------------+---------------------------------------+\n|                      RAM                     |                 VRAM 0                |\n+--------------------+------------+------------+----------------+----------+-----------+\n| LAYERS (I + T + O) |     UMA    |   NONUMA   | LAYERS (T + O) |    UMA   |   NONUMA  |\n+--------------------+------------+------------+----------------+----------+-----------+\n|      1 + 0 + 0     | 243.89 MiB | 393.89 MiB |     80 + 1     | 2.50 GiB | 28.92 GiB |\n+--------------------+------------+------------+----------------+----------+-----------+\n</code></pre> <p>From the output, we can see that the estimated VRAM requirement for this model exceeds the 22GB VRAM available on each worker node. Thus, we need to distribute the inference across multiple workers to successfully run the model.</p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<p>Follow these steps to deploy the model from Hugging Face, enabling distributed inference:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Enable the <code>GGUF</code> checkbox to filter models by GGUF format.</li> <li>Use the search bar in the top left to search for the model name <code>Qwen/Qwen2.5-72B-Instruct-GGUF</code>.</li> <li>In the <code>Available Files</code> section, select the <code>q2_k</code> quantization format.</li> <li>Expand the <code>Advanced</code> section and scroll down. Disable the <code>Allow CPU Offloading</code> option and verify that the <code>Allow Distributed Inference Across Workers</code> option is enabled(this is enabled by default). GPUStack will evaluate the available resources in the cluster and run the model in a distributed manner if required.</li> <li>Click the <code>Save</code> button to deploy the model.</li> </ol> <p></p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#step-2-verify-the-model-deployment","title":"Step 2: Verify the Model Deployment","text":"<p>Once the model is deployed, verify the deployment on the <code>Models</code> page, where you can view details about how the model is running across multiple workers.</p> <p></p> <p>You can also check worker and GPU resource usage by navigating to the <code>Resources</code> page.</p> <p></p> <p>Finally, go to the <code>Playground</code> page to interact with the model and verify that everything is functioning correctly.</p> <p></p>"},{"location":"tutorials/performing-distributed-inference-across-workers/#conclusion","title":"Conclusion","text":"<p>Congratulations! You have successfully configured and run distributed inference across multiple workers using GPUStack.</p>"},{"location":"tutorials/running-inference-with-ascend-npus/","title":"Running Inference With Ascend NPUs","text":"<p>GPUStack supports running inference on Ascend NPUs. This tutorial will guide you through the configuration steps.</p>"},{"location":"tutorials/running-inference-with-ascend-npus/#system-and-hardware-support","title":"System and Hardware Support","text":"OS Status Verified Linux Support Ubuntu 20.04 Device Status Verified Ascend 910 Support Ascend 910B"},{"location":"tutorials/running-inference-with-ascend-npus/#setup-steps","title":"Setup Steps","text":""},{"location":"tutorials/running-inference-with-ascend-npus/#install-ascend-packages","title":"Install Ascend packages","text":"<ol> <li>Download Ascend packages</li> </ol> <p>Choose the packages according to your system, hardware and GPUStack is compatible with CANN 8.x from resources download center(links below).</p> <p>Download the driver and firmware from here.</p> Package Name Description Ascend-hdk-{chiptype}-npu-driver{version}_linux-{arch}.run Ascend Driver (run format) Ascend-hdk-{chiptype}-npu-firmware{version}.run Ascend (run format) <p>Download the toolkit and kernels from here.</p> Package Name Description Ascend-cann-toolkit_{version}_linux-{arch}.run CANN Toolkit (run format) Ascend-cann-kernels-{chiptype}{version}_linux-{arch}.run CANN Kernels (run format) <ol> <li>Create the user and group for running</li> </ol> <pre><code>sudo groupadd -g HwHiAiUser\nsudo useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash\nsudo usermod -aG HwHiAiUser $USER\n</code></pre> <ol> <li>Install driver</li> </ol> <pre><code>sudo chmod +x Ascend-hdk-xxx-npu-driver_x.x.x_linux-{arch}.run\n# Driver installation, default installation path: \"/usr/local/Ascend\"\nsudo sh Ascend-hdk-xxx-npu-driver_x.x.x_linux-{arch}.run --full --install-for-all\n</code></pre> <p>If you see the following message, the firmware installation is complete:</p> <pre><code>Driver package installed successfully!\n</code></pre> <ol> <li>Verify successful driver installation</li> </ol> <p>After the driver successful installation, run the <code>npu-smi info</code> command to check if the driver was installed correctly.</p> <pre><code>$npu-smi info\n+------------------------------------------------------------------------------------------------+\n| npu-smi 23.0.1                   Version: 23.0.1                                               |\n+---------------------------+---------------+----------------------------------------------------+\n| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|\n| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |\n+===========================+===============+====================================================+\n| 4     910B3               | OK            | 93.6        40                0    / 0             |\n| 0                         | 0000:01:00.0  | 0           0    / 0          3161 / 65536         |\n+===========================+===============+====================================================+\n+---------------------------+---------------+----------------------------------------------------+\n| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |\n+===========================+===============+====================================================+\n| No running processes found in NPU 4                                                            |\n+===========================+===============+====================================================+\n</code></pre> <ol> <li>Install firmware</li> </ol> <pre><code>sudo chmod +x Ascend-hdk-xxx-npu-firmware_x.x.x.x.X.run\nsudo sh Ascend-hdk-xxx-npu-firmware_x.x.x.x.X.run --full\n</code></pre> <p>If you see the following message, the firmware installation is complete:</p> <pre><code>Firmware package installed successfully!\n</code></pre> <ol> <li>Install toolkit and kernels</li> </ol> <p>As an example for Ubuntu, adapt commands according to your system.</p> <p>Check for dependencies to ensure Python, GCC, and other required tools are installed.</p> <pre><code>gcc --version\ng++ --version\nmake --version\ncmake --version\ndpkg -l zlib1g| grep zlib1g| grep ii\ndpkg -l zlib1g-dev| grep zlib1g-dev| grep ii\ndpkg -l libsqlite3-dev| grep libsqlite3-dev| grep ii\ndpkg -l openssl| grep openssl| grep ii\ndpkg -l libssl-dev| grep libssl-dev| grep ii\ndpkg -l libffi-dev| grep libffi-dev| grep ii\ndpkg -l libbz2-dev| grep libbz2-dev| grep ii\ndpkg -l libxslt1-dev| grep libxslt1-dev| grep ii\ndpkg -l unzip| grep unzip| grep ii\ndpkg -l pciutils| grep pciutils| grep ii\ndpkg -l net-tools| grep net-tools| grep ii\ndpkg -l libblas-dev| grep libblas-dev| grep ii\ndpkg -l gfortran| grep gfortran| grep ii\ndpkg -l libblas3| grep libblas3| grep ii\n</code></pre> <p>If the commands return messages showing missing packages, install them as follows (adjust the command if only specific packages are missing):</p> <pre><code>sudo apt-get install -y gcc g++ make cmake zlib1g zlib1g-dev openssl libsqlite3-dev libssl-dev libffi-dev libbz2-dev libxslt1-dev unzip pciutils net-tools libblas-dev gfortran libblas3\n</code></pre> <p>Install Python dependencies:</p> <pre><code>pip3 install --upgrade pip\npip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions\n</code></pre> <p>Install the toolkit and kernels:</p> <pre><code>chmod +x Ascend-cann-toolkit_{vesion}_linux-{arch}.run\nchmod +x Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run\n\nsh Ascend-cann-toolkit_{vesion}_linux-{arch}.run --install\nsh Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run --install\n</code></pre> <p>Once installation completes, you should see a success message like this:</p> <pre><code>xxx install success\n</code></pre> <ol> <li>Configure environment variables</li> </ol> <pre><code>echo \"source ~/Ascend/ascend-toolkit/set_env.sh\" &gt;&gt; ~/.bashrc\nsource ~/.bashrc\n</code></pre> <p>For more details, refer to the Ascend Documentation.</p>"},{"location":"tutorials/running-inference-with-ascend-npus/#installing-gpustack","title":"Installing GPUStack","text":"<p>Once your environment is ready, you can install GPUStack following the installation guide.</p> <p>Once installed, you should see that GPUStack successfully recognizes the Ascend Device in the resources page.</p> <p></p>"},{"location":"tutorials/running-inference-with-ascend-npus/#running-inference","title":"Running Inference","text":"<p>After installation, you can deploy models and run inference. Refer to the model management for usage details.</p> <p>The Ascend NPU supports inference through the llama-box (llama.cpp) backend. For supported models, see the llama.cpp Ascend NPU model supports.</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/","title":"Running Inference With Moore Threads GPUs","text":"<p>GPUStack supports running inference on Moore Threads GPUs. This tutorial provides a comprehensive guide to configuring your system for optimal performance.</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#system-and-hardware-support","title":"System and Hardware Support","text":"OS Architecture Status Verified Linux x86_64 Support Ubuntu 20.04/22.04 Device Status Verified MTT S80 Support Yes MTT S3000 Support Yes MTT S4000 Support Yes"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#prerequisites","title":"Prerequisites","text":"<p>The following instructions are applicable for <code>Ubuntu 20.04/22.04</code> systems with <code>x86_64</code> architecture.</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#configure-the-container-runtime","title":"Configure the Container Runtime","text":"<p>Follow these links to install and configure the container runtime:</p> <ul> <li>Install Docker: Docker Installation Guide</li> <li>Install the latest drivers for MTT S80/S3000/S4000 (currently rc3.1.0): MUSA SDK Download</li> <li>Install the MT Container Toolkits (currently v1.9.0): MT CloudNative Toolkits Download</li> </ul>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#verify-container-runtime-configuration","title":"Verify Container Runtime Configuration","text":"<p>Ensure the output shows the default runtime as <code>mthreads</code>.</p> <pre><code>$ (cd /usr/bin/musa &amp;&amp; sudo ./docker setup $PWD)\n$ docker info | grep mthreads\n Runtimes: mthreads mthreads-experimental runc\n Default Runtime: mthreads\n</code></pre>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#installing-gpustack","title":"Installing GPUStack","text":"<p>To set up an isolated environment for GPUStack, we recommend using Docker.</p> <pre><code>docker run -d --name gpustack-musa -p 9009:80 --ipc=host -v gpustack-data:/var/lib/gpustack \\\n    gpustack/gpustack:main-musa\n</code></pre> <p>This command will:</p> <ul> <li>Start a container with the GPUStack image.</li> <li>Expose the GPUStack web interface on port <code>9009</code>.</li> <li>Mount the <code>gpustack-data</code> volume to store the GPUStack data.</li> </ul> <p>To check the logs of the running container, use the following command:</p> <pre><code>docker logs -f gpustack-musa\n</code></pre> <p>If the following message appears, the GPUStack container is running successfully:</p> <pre><code>2024-11-15T23:37:46+00:00 - gpustack.server.server - INFO - Serving on 0.0.0.0:80.\n2024-11-15T23:37:46+00:00 - gpustack.worker.worker - INFO - Starting GPUStack worker.\n</code></pre> <p>Once the container is running, access the GPUStack web interface by navigating to <code>http://localhost:9009</code> in your browser.</p> <p>After the initial setup for GPUStack, you should see the following screen:</p>"},{"location":"tutorials/running-inference-with-moorethreads-gpus/#dashboard","title":"Dashboard","text":""},{"location":"tutorials/running-inference-with-moorethreads-gpus/#workers","title":"Workers","text":""},{"location":"tutorials/running-inference-with-moorethreads-gpus/#gpus","title":"GPUs","text":""},{"location":"tutorials/running-inference-with-moorethreads-gpus/#running-inference","title":"Running Inference","text":"<p>After installation, you can deploy models and run inference. Refer to the model management for detailed usage instructions.</p> <p>Moore Threads GPUs support inference through the llama-box (llama.cpp) backend. Most recent models are supported (e.g., llama3.2:1b, llama3.2-vision:11b, qwen2.5:7b, etc.).</p> <p>Use <code>mthreads-gmi</code> to verify if the model is offloaded to the GPU.</p> <pre><code>root@a414c45864ee:/# mthreads-gmi\nSat Nov 16 12:00:16 2024\n---------------------------------------------------------------\n    mthreads-gmi:1.14.0          Driver Version:2.7.0\n---------------------------------------------------------------\nID   Name           |PCIe                |%GPU  Mem\n     Device Type    |Pcie Lane Width     |Temp  MPC Capable\n                                         |      ECC Mode\n+-------------------------------------------------------------+\n0    MTT S80        |00000000:01:00.0    |98%   1339MiB(16384MiB)\n     Physical       |16x(16x)            |56C   YES\n                                         |      N/A\n---------------------------------------------------------------\n\n---------------------------------------------------------------\nProcesses:\nID   PID       Process name                         GPU Memory\n                                                         Usage\n+-------------------------------------------------------------+\n0    120       ...ird_party/bin/llama-box/llama-box       2MiB\n0    2022      ...ird_party/bin/llama-box/llama-box    1333MiB\n---------------------------------------------------------------\n</code></pre>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/","title":"Running Inference on Copilot+ PCs with Snapdragon X","text":"<p>GPUStack supports running on ARM64 Windows, enabling use on Snapdragon X-based Copilot+ PCs.</p> <p>Note</p> <p>Only CPU-based inference is supported on Snapdragon X devices. GPUStack does not currently support GPU or NPU acceleration on this platform.</p>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#prerequisites","title":"Prerequisites","text":"<ul> <li>A Copilot+ PC with Snapdragon X. In this tutorial, we use the Dell XPS 13 9345.</li> <li>Install AMD64 Python (version 3.10 to 3.12). See details</li> </ul>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#installing-gpustack","title":"Installing GPUStack","text":"<p>Run PowerShell as administrator (avoid using PowerShell ISE), then run the following command to install GPUStack:</p> <pre><code>Invoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>After installation, follow the on-screen instructions to obtain credentials and log in to the GPUStack UI.</p>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#deploying-a-model","title":"Deploying a Model","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button and select <code>Ollama Library</code> from the dropdown.</li> <li>Enter <code>llama3.2</code> in the <code>Name</code> field.</li> <li>Select <code>llama3.2</code> from the <code>Ollama Model</code> dropdown.</li> <li>Click <code>Save</code> to deploy the model.</li> </ol> <p>Once deployed, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/#running-inference","title":"Running Inference","text":"<p>Navigate to the <code>Playground</code> page in the GPUStack UI, where you can interact with the deployed model.</p> <p></p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/","title":"Setting Up a Multi-node GPUStack Cluster","text":"<p>This tutorial will guide you through setting up a multi-node GPUStack cluster, where you can distribute your workloads across multiple GPU-enabled nodes. This guide assumes you have basic knowledge of running commands on Linux, macOS, or Windows systems.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#prerequisites","title":"Prerequisites","text":"<p>Before starting, ensure you have the following:</p> <ul> <li>Multiple nodes with supported OS and GPUs for GPUStack installation. View supported platforms and supported accelerators for more information.</li> <li>Nodes are connected to the same network and can communicate with each other.</li> </ul>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-1-install-gpustack-on-the-server-node","title":"Step 1: Install GPUStack on the Server Node","text":"<p>First, you need to install GPUStack on one of the nodes to act as the server node. Follow the instructions below based on your operating system.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#linux-or-macos","title":"Linux or macOS","text":"<p>Run the following command on your server node:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s -\n</code></pre>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#windows","title":"Windows","text":"<p>Run PowerShell as administrator and execute the following command:</p> <pre><code>Invoke-Expression (Invoke-WebRequest -Uri \"https://get.gpustack.ai\" -UseBasicParsing).Content\n</code></pre> <p>Once GPUStack is installed, you can proceed to configure your cluster by adding worker nodes.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-2-retrieve-the-token-from-the-server-node","title":"Step 2: Retrieve the Token from the Server Node","text":"<p>To add worker nodes to the cluster, you need the token generated by GPUStack on the server node. On the server node, run the following command to get the token:</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#linux-or-macos_1","title":"Linux or macOS","text":"<pre><code>cat /var/lib/gpustack/token\n</code></pre>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#windows_1","title":"Windows","text":"<pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\token\" -Raw\n</code></pre> <p>This token will be required in the next steps to authenticate worker nodes.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-3-add-worker-nodes-to-the-cluster","title":"Step 3: Add Worker Nodes to the Cluster","text":"<p>Now, you will install GPUStack on additional nodes (worker nodes) and connect them to the server node using the token.</p> <p>Linux or macOS Worker Nodes</p> <p>Run the following command on each worker node, replacing http://myserver with the URL of your server node and mytoken with the token retrieved in Step 2:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s - --server-url http://myserver --token mytoken\n</code></pre> <p>Windows Worker Nodes</p> <p>Run PowerShell as administrator on each worker node and use the following command, replacing http://myserver and mytoken with the server URL and token:</p> <pre><code>Invoke-Expression \"&amp; { $((Invoke-WebRequest -Uri 'https://get.gpustack.ai' -UseBasicParsing).Content) } --server-url http://myserver --token mytoken\"\n</code></pre> <p>Once the command is executed, each worker node will connect to the main server and become part of the GPUStack cluster.</p>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#step-4-verify-the-cluster-setup","title":"Step 4: Verify the Cluster Setup","text":"<p>After adding the worker nodes, you can verify that the cluster is set up correctly by accessing the GPUStack UI.</p> <ol> <li>Open a browser and navigate to <code>http://myserver</code> (replace myserver with the actual server URL).</li> <li>Log in with the default credentials (username <code>admin</code>). To retrieve the default password, run the following command on the server node:</li> </ol> <p>Linux or macOS</p> <pre><code>cat /var/lib/gpustack/initial_admin_password\n</code></pre> <p>Windows</p> <pre><code>Get-Content -Path \"$env:APPDATA\\gpustack\\initial_admin_password\" -Raw\n</code></pre> <ol> <li>After logging in, navigate to the <code>Resources</code> page in the UI to see all connected nodes and their GPUs. You should see your worker nodes listed and ready for serving LLMs.</li> </ol>"},{"location":"tutorials/setting-up-a-multi-node-gpustack-cluster/#conclusion","title":"Conclusion","text":"<p>Congratulations! You've successfully set up a multi-node GPUStack cluster! You can now scale your workloads across multiple nodes, making full use of your available GPUs to handle your tasks efficiently.</p>"},{"location":"tutorials/using-audio-models/","title":"Using Audio Models","text":"<p>GPUStack supports running both speech-to-text and text-to-speech models. Speech-to-text models convert audio inputs in various languages into written text, while text-to-speech models transform written text into natural and expressive speech.</p> <p>In this tutorial, we will walk you through deploying and using speech-to-text and text-to-speech models in GPUStack.</p>"},{"location":"tutorials/using-audio-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>A Linux system with AMD architecture or macOS.</li> <li>Access to Hugging Face for downloading the model files.</li> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> </ul>"},{"location":"tutorials/using-audio-models/#running-speech-to-text-model","title":"Running Speech-to-Text Model","text":""},{"location":"tutorials/using-audio-models/#step-1-deploy-speech-to-text-model","title":"Step 1: Deploy Speech-to-Text Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar in the top left to search for the model name <code>Systran/faster-whisper-medium</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-audio-models/#step-2-interact-with-speech-to-text-model-models","title":"Step 2: Interact with Speech-to-Text Model Models","text":"<ol> <li>Navigate to the <code>Playground</code> &gt; <code>Audio</code> page in the GPUStack UI.</li> <li>Select the <code>Speech to Text</code> Tab.</li> <li>Select the deployed model from the top-right dropdown.</li> <li>Click the <code>Upload</code> button to upload audio file or click the <code>Microphone</code> button to record audio.</li> <li>Click the <code>Generate Text Content</code> button to generate the text.</li> </ol>"},{"location":"tutorials/using-audio-models/#running-text-to-speech-model","title":"Running Text-to-Speech Model","text":""},{"location":"tutorials/using-audio-models/#step-1-deploy-text-to-speech-model","title":"Step 1: Deploy Text-to-Speech Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar in the top left to search for the model name <code>FunAudioLLM/CosyVoice-300M</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-audio-models/#step-2-interact-with-text-to-speech-model-models","title":"Step 2: Interact with Text to Speech Model Models","text":"<ol> <li>Navigate to the <code>Playground</code> &gt; <code>Audio</code> page in the GPUStack UI.</li> <li>Select the <code>Text to Speech</code> Tab.</li> <li>Choose the deployed model from the dropdown menu in the top-right corner. Then, configure the voice and output audio format.</li> <li>Input the text to generate.</li> <li>Click the <code>Submit</code> button to generate the text.</li> </ol>"},{"location":"tutorials/using-image-generation-models/","title":"Using Image Generation Models","text":"<p>GPUStack supports deploying and running state-of-the-art image generation models. These models allow you to generate stunning images from textual descriptions, enabling applications in design, content creation, and more.</p> <p>In this tutorial, we will walk you through deploying and using image generation models in GPUStack.</p>"},{"location":"tutorials/using-image-generation-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>A GPU that has at least 12 GB of VRAM.</li> <li>Access to Hugging Face for downloading the model files.</li> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> </ul>"},{"location":"tutorials/using-image-generation-models/#step-1-deploy-the-stable-diffusion-model","title":"Step 1: Deploy the Stable Diffusion Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Use the search bar in the top left to search for the model name <code>gpustack/stable-diffusion-v3-5-medium-GGUF</code>.</li> <li>In the <code>Available Files</code> section, select the <code>stable-diffusion-v3-5-medium-Q4_0.gguf</code> file.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-image-generation-models/#step-2-use-the-model-for-image-generation","title":"Step 2: Use the Model for Image Generation","text":"<ol> <li>Navigate to the <code>Playground</code> &gt; <code>Image</code> page in the GPUStack UI.</li> <li>Verify that the deployed model is selected from the top-right <code>Model</code> dropdown.</li> <li>Enter a prompt describing the image you want to generate. For example:</li> </ol> <pre><code>a female character with long, flowing hair that appears to be made of ethereal, swirling patterns resembling the Northern Lights or Aurora Borealis. The background is dominated by deep blues and purples, creating a mysterious and dramatic atmosphere. The character's face is serene, with pale skin and striking features. She wears a dark-colored outfit with subtle patterns. The overall style of the artwork is reminiscent of fantasy or supernatural genres.\n</code></pre> <ol> <li>Select <code>euler</code> in the <code>Sampler</code> dropdown.</li> <li>Set the <code>Sample Steps</code> to <code>20</code>.</li> <li>Click the <code>Submit</code> button to create the image.</li> </ol> <p>The generated image will be displayed in the UI. Your image may look different given the seed and randomness involved in the generation process.</p> <p></p>"},{"location":"tutorials/using-image-generation-models/#conclusion","title":"Conclusion","text":"<p>Congratulations! You\u2019ve successfully deployed and used an image generation model in GPUStack. With this setup, you can generate unique and visually compelling images from textual prompts. Experiment with different prompts and settings to push the boundaries of what\u2019s possible.</p>"},{"location":"tutorials/using-reranker-models/","title":"Using Reranker Models","text":"<p>Reranker models are specialized models designed to improve the ranking of a list of items based on relevance to a given query. They are commonly used in information retrieval and search systems to refine initial search results, prioritizing items that are more likely to meet the user\u2019s intent. Reranker models take the initial document list and reorder items to enhance precision in applications such as search engines, recommendation systems, and question-answering tasks.</p> <p>In this tutorial, we will guide you through deploying and using reranker models in GPUStack.</p>"},{"location":"tutorials/using-reranker-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>GPUStack is installed and running. If not, refer to the Quickstart Guide.</li> <li>Access to Hugging Face for downloading the model files.</li> </ul>"},{"location":"tutorials/using-reranker-models/#step-1-deploy-the-model","title":"Step 1: Deploy the Model","text":"<p>Follow these steps to deploy the model from Hugging Face:</p> <ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click the <code>Deploy Model</code> button.</li> <li>In the dropdown, select <code>Hugging Face</code> as the source for your model.</li> <li>Enable the <code>GGUF</code> checkbox to filter models by GGUF format.</li> <li>Use the search bar in the top left to search for the model name <code>gpustack/bge-reranker-v2-m3-GGUF</code>.</li> <li>Leave everything as default and click the <code>Save</code> button to deploy the model.</li> </ol> <p></p> <p>After deployment, you can monitor the model's status on the <code>Models</code> page.</p> <p></p>"},{"location":"tutorials/using-reranker-models/#step-2-generate-an-api-key","title":"Step 2: Generate an API Key","text":"<p>We will use the GPUStack API to interact with the model. To do this, you need to generate an API key:</p> <ol> <li>Navigate to the <code>API Keys</code> page in the GPUStack UI.</li> <li>Click the <code>New API Key</code> button.</li> <li>Enter a name for the API key and click the <code>Save</code> button.</li> <li>Copy the generated API key. You can only view the API key once, so make sure to save it securely.</li> </ol>"},{"location":"tutorials/using-reranker-models/#step-3-reranking","title":"Step 3: Reranking","text":"<p>With the model deployed and an API key, you can rerank a list of documents via the GPUStack API. Here is an example script using <code>curl</code>:</p> <pre><code>export SERVER_URL=&lt;your-server-url&gt;\nexport GPUSTACK_API_KEY=&lt;your-api-key&gt;\ncurl $SERVER_URL/v1/rerank \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -d '{\n        \"model\": \"bge-reranker-v2-m3\",\n        \"query\": \"What is a panda?\",\n        \"top_n\": 3,\n        \"documents\": [\n            \"hi\",\n            \"it is a bear\",\n            \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n        ]\n    }' | jq\n</code></pre> <p>Replace <code>&lt;your-server-url&gt;</code> with the URL of your GPUStack server and <code>&lt;your-api-key&gt;</code> with the API key you generated in the previous step.</p> <p>Example response:</p> <pre><code>{\n  \"model\": \"bge-reranker-v2-m3\",\n  \"object\": \"list\",\n  \"results\": [\n    {\n      \"document\": {\n        \"text\": \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n      },\n      \"index\": 2,\n      \"relevance_score\": 1.951932668685913\n    },\n    {\n      \"document\": {\n        \"text\": \"it is a bear\"\n      },\n      \"index\": 1,\n      \"relevance_score\": -3.7347371578216553\n    },\n    {\n      \"document\": {\n        \"text\": \"hi\"\n      },\n      \"index\": 0,\n      \"relevance_score\": -6.157620906829834\n    }\n  ],\n  \"usage\": {\n    \"prompt_tokens\": 69,\n    \"total_tokens\": 69\n  }\n}\n</code></pre>"},{"location":"tutorials/using-vision-language-models/","title":"Using Vision Language Models","text":"<p>Vision Language Models can process both visual (image) and language (text) data simultaneously, making them versatile tools for various applications, such as image captioning, visual question answering, and more. In this tutorial, you will learn how to deploy and interact with Vision Language Models (VLMs) in GPUStack.</p> <p>The procedure for deploying and interacting with these models in GPUStack is similar. The main difference is the parameters you need to set when deploying the models. For more information on the parameters you can set, please refer to Backend Parameters .</p> <p>In this tutorial, we will cover the deployment of the following models:</p> <ul> <li>Llama3.2-Vision</li> <li>Qwen2-VL</li> <li>Pixtral</li> <li>Phi3.5-Vision</li> </ul>"},{"location":"tutorials/using-vision-language-models/#prerequisites","title":"Prerequisites","text":"<p>Before you begin, ensure that you have the following:</p> <ul> <li>A Linux machine with one or more GPUs that has at least 30 GB of VRAM in total. We will use the vLLM backend which only supports Linux.</li> <li>Access to Hugging Face and a Hugging Face API key for downloading the model files.</li> <li>You have been granted access to the above models on Hugging Face. Llama3.2-VL and Pixtral are gated models, and you need to request access to them.</li> </ul> <p>Note</p> <p>An Ubuntu node equipped with one H100 (80GB) GPU is used throughout this tutorial.</p>"},{"location":"tutorials/using-vision-language-models/#step-1-install-gpustack","title":"Step 1: Install GPUStack","text":"<p>Run the following command to install GPUStack:</p> <pre><code>curl -sfL https://get.gpustack.ai | sh -s - --huggingface-token &lt;Hugging Face API Key&gt;\n</code></pre> <p>Replace <code>&lt;Hugging Face API Key&gt;</code> with your Hugging Face API key. GPUStack will use this key to download the model files.</p>"},{"location":"tutorials/using-vision-language-models/#step-2-log-in-to-gpustack-ui","title":"Step 2: Log in to GPUStack UI","text":"<p>Run the following command to get the default password:</p> <pre><code>cat /var/lib/gpustack/initial_admin_password\n</code></pre> <p>Open your browser and navigate to <code>http://&lt;your-server-ip&gt;</code>. Replace <code>&lt;your-server-ip&gt;</code> with the IP address of your server. Log in using the username <code>admin</code> and the password you obtained in the previous step.</p>"},{"location":"tutorials/using-vision-language-models/#step-3-deploy-vision-language-models","title":"Step 3: Deploy Vision Language Models","text":""},{"location":"tutorials/using-vision-language-models/#deploy-llama32-vision","title":"Deploy Llama3.2-Vision","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>meta-llama/Llama-3.2-11B-Vision-Instruct</code> in the search bar.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button multiple times and add the following parameters:</li> </ol> <ul> <li><code>--enforce-eager</code></li> <li><code>--max-num-seqs=16</code></li> <li><code>--max-model-len=8192</code></li> </ul> <ol> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#deploy-qwen2-vl","title":"Deploy Qwen2-VL","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>Qwen/Qwen2-VL-7B-Instruct</code> in the search bar.</li> <li>Click the <code>Save</code> button. The default configurations should work as long as you have enough GPU resources.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#deploy-pixtral","title":"Deploy Pixtral","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>mistralai/Pixtral-12B-2409</code> in the search bar.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button multiple times and add the following parameters:</li> </ol> <ul> <li><code>--tokenizer-mode=mistral</code></li> <li><code>--limit-mm-per-prompt=image=4</code></li> </ul> <ol> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#deploy-phi35-vision","title":"Deploy Phi3.5-Vision","text":"<ol> <li>Navigate to the <code>Models</code> page in the GPUStack UI.</li> <li>Click on the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</li> <li>Search for <code>microsoft/Phi-3.5-vision-instruct</code> in the search bar.</li> <li>Expand the <code>Advanced</code> section in configurations and scroll down to the <code>Backend Parameters</code> section.</li> <li>Click on the <code>Add Parameter</code> button and add the following parameter:</li> </ol> <ul> <li><code>--trust-remote-code</code></li> </ul> <ol> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#step-4-interact-with-vision-language-models","title":"Step 4: Interact with Vision Language Models","text":"<ol> <li>Navigate to the <code>Playground</code> page in the GPUStack UI.</li> <li>Select the deployed model from the top-right dropdown.</li> <li>Click on the <code>Upload Image</code> button above the input text area and upload an image.</li> <li>Enter a prompt in the input text area. For example, \"Describe the image.\"</li> <li>Click the <code>Submit</code> button to generate the output.</li> </ol>"},{"location":"tutorials/using-vision-language-models/#conclusion","title":"Conclusion","text":"<p>In this tutorial, you learned how to deploy and interact with Vision Language Models in GPUStack. You can use the same approach to deploy other Vision Language Models not covered in this tutorial. If you have any questions or need further assistance, feel free to reach out to us.</p>"},{"location":"user-guide/api-key-management/","title":"API Key Management","text":"<p>GPUStack supports authentication using API keys. Each GPUStack user can generate and manage their own API keys.</p>"},{"location":"user-guide/api-key-management/#create-api-key","title":"Create API Key","text":"<ol> <li>Navigate to the <code>API Keys</code> page.</li> <li>Click the <code>New API Key</code> button.</li> <li>Fill in the <code>Name</code>, <code>Description</code>, and select the <code>Expiration</code> of the API key.</li> <li>Click the <code>Save</code> button.</li> <li>Copy and store the key somewhere safe, then click the <code>Done</code> button.</li> </ol> <p>Note</p> <p>Please note that you can only see the generated API key once upon creation.</p>"},{"location":"user-guide/api-key-management/#delete-api-key","title":"Delete API Key","text":"<ol> <li>Navigate to the <code>API Keys</code> page.</li> <li>Find the API key you want to delete.</li> <li>Click the <code>Delete</code> button in the <code>Operations</code> column.</li> <li>Confirm the deletion.</li> </ol>"},{"location":"user-guide/api-key-management/#use-api-key","title":"Use API Key","text":"<p>GPUStack supports using the API key as a bearer token. The following is an example using curl:</p> <pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -d '{\n    \"model\": \"llama3\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a helpful assistant.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n      }\n    ],\n    \"stream\": true\n  }'\n</code></pre>"},{"location":"user-guide/image-generation-apis/","title":"Image Generation APIs","text":"<p>GPUStack provides APIs for generating images given a prompt and/or an input image when running diffusion models.</p> <p>Note</p> <p>The image generation APIs are only available when using the llama-box inference backend.</p>"},{"location":"user-guide/image-generation-apis/#supported-models","title":"Supported Models","text":"<p>The following models are available for image generation:</p> <p>Tip</p> <p>Please use the converted GGUF models provided by GPUStack. Check the model link for more details.</p> <ul> <li>stabilityai/stable-diffusion-3.5-large-turbo</li> <li>stabilityai/stable-diffusion-3.5-large</li> <li>stabilityai/stable-diffusion-3.5-medium</li> <li>stabilityai/stable-diffusion-3-medium</li> <li>TencentARC/FLUX.1-mini</li> <li>Freepik/FLUX.1-lite</li> <li>black-forest-labs/FLUX.1-dev</li> <li>black-forest-labs/FLUX.1-schnell</li> <li>stabilityai/sdxl-turbo</li> <li>stabilityai/stable-diffusion-xl-refiner-1.0</li> <li>stabilityai/stable-diffusion-xl-base-1.0</li> <li>stabilityai/sd-turbo</li> <li>stabilityai/stable-diffusion-2-1</li> <li>stable-diffusion-v1-5/stable-diffusion-v1-5</li> <li>CompVis/stable-diffusion-v1-4</li> </ul>"},{"location":"user-guide/image-generation-apis/#api-details","title":"API Details","text":"<p>The image generation APIs adhere to OpenAI API specification. While OpenAI APIs for image generation are simple and opinionated, GPUStack extends these capabilities with additional features.</p>"},{"location":"user-guide/image-generation-apis/#create-image","title":"Create Image","text":""},{"location":"user-guide/image-generation-apis/#streaming","title":"Streaming","text":"<p>This image generation API supports streaming responses to return the progressing of the generation. To enable streaming, set the <code>stream</code> parameter to <code>true</code> in the request body. Example:</p> <pre><code>REQUEST : (application/json)\n{\n  \"n\": 1,\n  \"response_format\": \"b64_json\",\n  \"size\": \"512x512\",\n  \"prompt\": \"A lovely cat\",\n  \"quality\": \"standard\",\n  \"stream\": true,\n  \"stream_options\": {\n    \"include_usage\": true, // return usage information\n  }\n}\n\nRESPONSE : (text/event-stream)\ndata: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\ndata: [DONE]\n</code></pre>"},{"location":"user-guide/image-generation-apis/#advanced-options","title":"Advanced Options","text":"<p>This image generation API supports additional options to control the generation process. The following options are available:</p> <pre><code>REQUEST : (application/json)\n{\n  \"n\": 1,\n  \"response_format\": \"b64_json\",\n  \"size\": \"512x512\",\n  \"prompt\": \"A lovely cat\",\n  \"sampler\": \"euler\",      // required, select from euler_a;euler;heun;dpm2;dpm++2s_a;dpm++2m;dpm++2mv2;ipndm;ipndm_v;lcm\n  \"schedule\": \"default\",   // optional, select from default;discrete;karras;exponential;ays;gits\n  \"seed\": null,            // optional, random seed\n  \"cfg_scale\": 4.5,        // optional, for sampler, the scale of classifier-free guidance in the output phase\n  \"sample_steps\": 20,      // optional, number of sample steps\n  \"negative_prompt\": \"\",   // optional, negative prompt\n  \"stream\": true,\n  \"stream_options\": {\n    \"include_usage\": true, // return usage information\n  }\n}\n\nRESPONSE : (text/event-stream)\ndata: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n...\ndata: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\ndata: [DONE]\n</code></pre>"},{"location":"user-guide/image-generation-apis/#create-image-edit","title":"Create Image Edit","text":""},{"location":"user-guide/image-generation-apis/#streaming_1","title":"Streaming","text":"<p>This image generation API supports streaming responses to return the progressing of the generation. To enable streaming, set the <code>stream</code> parameter to <code>true</code> in the request body. Example:</p> <pre><code>REQUEST: (multipart/form-data)\nn=1\nresponse_format=b64_json\nsize=512x512\nprompt=\"A lovely cat\"\nquality=standard\nimage=...                         // required\nmask=...                          // optional\nstream=true\nstream_options_include_usage=true // return usage information\n\nRESPONSE : (text/event-stream)\nCASE 1: correct input image\n  data: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\n  data: [DONE]\nCASE 2: illegal input image\n  error: {\"code\": 400, \"message\": \"Invalid image\", \"type\": \"invalid_request_error\"}\n</code></pre>"},{"location":"user-guide/image-generation-apis/#advanced-options_1","title":"Advanced Options","text":"<p>This image generation API supports additional options to control the generation process. The following options are available:</p> <pre><code>REQUEST: (multipart/form-data)\nn=1\nresponse_format=b64_json\nsize=512x512\nprompt=\"A lovely cat\"\nimage=...                         // required\nmask=...                          // optional\nsampler=euler                     // required, select from euler_a;euler;heun;dpm2;dpm++2s_a;dpm++2m;dpm++2mv2;ipndm;ipndm_v;lcm\nschedule=default                  // optional, select from default;discrete;karras;exponential;ays;gits\nseed=null                         // optional, random seed\ncfg_scale=4.5                     // optional, for sampler, the scale of classifier-free guidance in the output phase\nsample_steps=20                   // optional, number of sample steps\nnegative_prompt=\"\"                // optional, negative prompt\nstream=true\nstream_options_include_usage=true // return usage information\n\nRESPONSE : (text/event-stream)\nCASE 1: correct input image\n  data: {\"created\":1731916353,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":10.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":50.0}], ...}\n  ...\n  data: {\"created\":1731916371,\"data\":[{\"index\":0,\"object\":\"image.chunk\",\"progress\":100.0,\"b64_json\":\"...\"}], \"usage\":{\"generation_per_second\":...,\"time_per_generation_ms\":...,\"time_to_process_ms\":...}, ...}\n  data: [DONE]\nCASE 2: illegal input image\n  error: {\"code\": 400, \"message\": \"Invalid image\", \"type\": \"invalid_request_error\"}\n</code></pre>"},{"location":"user-guide/image-generation-apis/#usage","title":"Usage","text":"<p>The followings are examples using the image generation APIs:</p>"},{"location":"user-guide/image-generation-apis/#curl-create-image","title":"curl (Create Image)","text":"<pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/image/generate \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -d '{\n        \"n\": 1,\n        \"response_format\": \"b64_json\",\n        \"size\": \"512x512\",\n        \"prompt\": \"A lovely cat\",\n        \"quality\": \"standard\",\n        \"stream\": true,\n        \"stream_options\": {\n        \"include_usage\": true\n        }\n    }'\n</code></pre>"},{"location":"user-guide/image-generation-apis/#curl-create-image-edit","title":"curl (Create Image Edit)","text":"<pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/image/edit \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -F image=\"@otter.png\" \\\n    -F mask=\"@mask.png\" \\\n    -F prompt=\"A lovely cat\" \\\n    -F n=1 \\\n    -F size=\"512x512\"\n</code></pre>"},{"location":"user-guide/inference-backends/","title":"Inference Backends","text":"<p>GPUStack supports the following inference backends:</p> <ul> <li>llama-box</li> <li>vLLM</li> <li>vox-box</li> </ul> <p>When users deploy a model, the backend is selected automatically based on the following criteria:</p> <ul> <li>If the model is a GGUF model, <code>llama-box</code> is used.</li> <li>If the model is a known <code>text-to-speech</code> or <code>speech-to-text</code> model, <code>vox-box</code> is used.</li> <li>Otherwise, <code>vLLM</code> is used.</li> </ul>"},{"location":"user-guide/inference-backends/#llama-box","title":"llama-box","text":"<p>llama-box is a LM inference server based on llama.cpp and stable-diffusion.cpp.</p>"},{"location":"user-guide/inference-backends/#supported-platforms","title":"Supported Platforms","text":"<p>The llama-box backend supports Linux, macOS and Windows (with CPU offloading only on Windows ARM architecture) platforms.</p>"},{"location":"user-guide/inference-backends/#supported-models","title":"Supported Models","text":"<ul> <li>LLMs: For supported LLMs, refer to the llama.cpp README.</li> <li>Difussion Models: Supported models are listed in this Hugging Face collection.</li> <li>Reranker Models: Supported models can be found in this Hugging Face collection.</li> </ul>"},{"location":"user-guide/inference-backends/#supported-features","title":"Supported Features","text":""},{"location":"user-guide/inference-backends/#allow-cpu-offloading","title":"Allow CPU Offloading","text":"<p>After enabling CPU offloading, GPUStack prioritizes loading as many layers as possible onto the GPU to optimize performance. If GPU resources are limited, some layers will be offloaded to the CPU, with full CPU inference used only when no GPU is available.</p>"},{"location":"user-guide/inference-backends/#allow-distributed-inference-across-workers","title":"Allow Distributed Inference Across Workers","text":"<p>Enable distributed inference across multiple workers. The primary Model Instance will communicate with backend instances on one or more others workers, offloading computation tasks to them.</p>"},{"location":"user-guide/inference-backends/#parameters-reference","title":"Parameters Reference","text":"<p>See the full list of supported parameters for llama-box here.</p>"},{"location":"user-guide/inference-backends/#vllm","title":"vLLM","text":"<p>vLLM is a high-throughput and memory-efficient LLMs inference engine. It is a popular choice for running LLMs in production. vLLM seamlessly supports most state-of-the-art open-source models, including: Transformer-like LLMs (e.g., Llama), Mixture-of-Expert LLMs (e.g., Mixtral), Embedding Models (e.g. E5-Mistral), Multi-modal LLMs (e.g., LLaVA)</p> <p>By default, GPUStack estimates the VRAM requirement for the model instance based on the model's metadata. You can customize the parameters to fit your needs. The following vLLM parameters might be useful:</p> <ul> <li><code>--gpu-memory-utilization</code> (default: 0.9): The fraction of GPU memory to use for the model instance.</li> <li><code>--max-model-len</code>: Model context length. For large-context models, GPUStack automatically sets this parameter to <code>8192</code> to simplify model deployment, especially in resource constrained environments. You can customize this parameter to fit your needs.</li> <li><code>--tensor-parallel-size</code>: Number of tensor parallel replicas. By default, GPUStack sets this parameter given the GPU resources available and the estimation of the model's memory requirement. You can customize this parameter to fit your needs.</li> </ul> <p>For more details, please refer to vLLM documentation.</p>"},{"location":"user-guide/inference-backends/#supported-platforms_1","title":"Supported Platforms","text":"<p>The vLLM backend works on AMD Linux.</p> <p>Note</p> <ol> <li>When users install GPUStack on amd64 Linux using the installation script, vLLM is automatically installed.</li> <li>When users deploy a model using the vLLM backend, GPUStack sets worker label selectors to <code>{\"os\": \"linux\", \"arch\": \"amd64\"}</code> by default to ensure the model instance is scheduled to proper workers. You can customize the worker label selectors in the model configuration.</li> </ol>"},{"location":"user-guide/inference-backends/#supported-models_1","title":"Supported Models","text":"<p>Please refer to the vLLM documentation for supported models.</p>"},{"location":"user-guide/inference-backends/#supported-features_1","title":"Supported Features","text":""},{"location":"user-guide/inference-backends/#multimodal-language-models","title":"Multimodal Language Models","text":"<p>vLLM supports multimodal language models listed here. When users deploy a vision language model using the vLLM backend, image inputs are supported in the chat completion API.</p>"},{"location":"user-guide/inference-backends/#parameters-reference_1","title":"Parameters Reference","text":"<p>See the full list of supported parameters for vLLM here.</p>"},{"location":"user-guide/inference-backends/#vox-box","title":"vox-box","text":"<p>vox-box is an inference engine designed for deploying text-to-speech and speech-to-text models. It also provides an API that is fully compatible with the OpenAI audio API.</p>"},{"location":"user-guide/inference-backends/#supported-platforms_2","title":"Supported Platforms","text":"<p>The vox-box backend supports Linux, macOS and Windows platforms.</p> <p>Note</p> <ol> <li>To use Nvidia GPUs, ensure the following NVIDIA libraries are installed on workers:<ul> <li>cuBLAS for CUDA 12</li> <li>cuDNN 9 for CUDA 12</li> </ul> </li> <li>When users install GPUStack on Linux, macOS and Windows using the installation script, vox-box is automatically installed.</li> <li>CosyVoice models are natively supported on Linux AMD architecture and macOS. However, these models are not supported on Linux ARM or Windows architectures.</li> </ol>"},{"location":"user-guide/inference-backends/#supported-models_2","title":"Supported Models","text":"Model Type Link Supported Platforms Faster-whisper-large-v3 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-large-v2 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-large-v1 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-medium speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-medium.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-small speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-small.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-distil-whisper-large-v3 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-distil-whisper-large-v2 speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-distil-whisper-medium.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-tiny speech-to-text Hugging Face, ModelScope Linux, macOS, Windows Faster-whisper-tiny.en speech-to-text Hugging Face, ModelScope Linux, macOS, Windows CosyVoice-300M-Instruct text-to-speech Hugging Face, ModelScope Linux(ARM not supported), macOS, Windows(Not supported) CosyVoice-300M-SFT text-to-speech Hugging Face, ModelScope Linux(ARM not supported), macOS, Windows(Not supported) CosyVoice-300M text-to-speech Hugging Face, ModelScope Linux(ARM not supported), macOS, Windows(Not supported) CosyVoice-300M-25Hz text-to-speech ModelScope Linux(ARM not supported), macOS, Windows(Not supported)"},{"location":"user-guide/inference-backends/#supported-features_2","title":"Supported Features","text":""},{"location":"user-guide/inference-backends/#allow-gpucpu-offloading","title":"Allow GPU/CPU Offloading","text":"<p>vox-box supports deploying models to NVIDIA GPUs. If GPU resources are insufficient, it will automatically deploy the models to the CPU.</p>"},{"location":"user-guide/model-management/","title":"Model Management","text":"<p>You can manage large language models in GPUStack by navigating to the <code>Models</code> page. A model in GPUStack contains one or multiple replicas of model instances. On deployment, GPUStack automatically computes resource requirements for the model instances from model metadata and schedules them to available workers accordingly.</p>"},{"location":"user-guide/model-management/#deploy-model","title":"Deploy Model","text":"<p>Currently, models from Hugging Face, ModelScope, Ollama and local paths are supported.</p>"},{"location":"user-guide/model-management/#deploying-a-hugging-face-model","title":"Deploying a Hugging Face Model","text":"<ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>Hugging Face</code> in the dropdown.</p> </li> <li> <p>Search the model by name from Hugging Face using the search bar in the top left. For example, <code>microsoft/Phi-3-mini-4k-instruct-gguf</code>. If you only want to search for GGUF models, check the \"GGUF\" checkbox.</p> </li> <li> <p>Select a file with the desired quantization format from <code>Available Files</code>.</p> </li> <li> <p>Adjust the <code>Name</code> and <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#deploying-a-modelscope-model","title":"Deploying a ModelScope Model","text":"<ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>ModelScope</code> in the dropdown.</p> </li> <li> <p>Search the model by name from ModelScope using the search bar in the top left. For example, <code>Qwen/Qwen2-0.5B-Instruct</code>. If you only want to search for GGUF models, check the \"GGUF\" checkbox.</p> </li> <li> <p>Select a file with the desired quantization format from <code>Available Files</code>.</p> </li> <li> <p>Adjust the <code>Name</code> and <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#deploying-an-ollama-model","title":"Deploying an Ollama Model","text":"<ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>Ollama Library</code> in the dropdown.</p> </li> <li> <p>Fill in the <code>Name</code> of the model.</p> </li> <li> <p>Select an <code>Ollama Model</code> from the dropdown list, or input any Ollama model you need. For example, <code>llama3</code>, <code>llama3:70b</code> or <code>youraccount/llama3:70b</code>.</p> </li> <li> <p>Adjust the <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#deploying-a-local-path-model","title":"Deploying a Local Path Model","text":"<p>You can deploy a model from a local path. The model path can be a directory (e.g., a downloaded Hugging Face model directory) or a file (e.g., a GGUF model file) located on workers. This is useful when running in an air-gapped environment.</p> <p>Note</p> <ol> <li>GPUStack does not check the validity of the model path for scheduling, which may lead to deployment failure if the model path is inaccessible. It is recommended to ensure the model path is accessible on all workers(e.g., using NFS, rsync, etc.). You can also use the worker selector configuration to deploy the model to specific workers.</li> <li>GPUStack cannot evaluate the model's resource requirements unless the server has access to the same model path. Consequently, you may observe empty VRAM/RAM allocations for a deployed model. To mitigate this, it is recommended to make the model files available on the same path on the server. Alternatively, you can customize backend parameters, such as <code>tensor-split</code>, to configure how the model is distributed across the GPUs.</li> </ol> <p>To deploy a local path model:</p> <ol> <li> <p>Click the <code>Deploy Model</code> button, then select <code>Local Path</code> in the dropdown.</p> </li> <li> <p>Fill in the <code>Name</code> of the model.</p> </li> <li> <p>Fill in the <code>Model Path</code>.</p> </li> <li> <p>Adjust the <code>Replicas</code> as needed.</p> </li> <li> <p>Expand the <code>Advanced</code> section for advanced configurations if needed. Please refer to the Advanced Model Configuration section for more details.</p> </li> <li> <p>Click the <code>Save</code> button.</p> </li> </ol>"},{"location":"user-guide/model-management/#edit-model","title":"Edit Model","text":"<ol> <li>Find the model you want to edit on the model list page.</li> <li>Click the <code>Edit</code> button in the <code>Operations</code> column.</li> <li>Update the attributes as needed. For example, change the <code>Replicas</code> to scale up or down.</li> <li>Click the <code>Save</code> button.</li> </ol> <p>Note</p> <p>After editing the model, the configuration will not be applied to existing model instances. You need to delete the existing model instances. GPUStack will recreate new instances based on the updated model configuration.</p>"},{"location":"user-guide/model-management/#delete-model","title":"Delete Model","text":"<ol> <li>Find the model you want to delete on the model list page.</li> <li>Click the ellipsis button in the <code>Operations</code> column, then select <code>Delete</code>.</li> <li>Confirm the deletion.</li> </ol>"},{"location":"user-guide/model-management/#view-model-instance","title":"View Model Instance","text":"<ol> <li>Find the model you want to check on the model list page.</li> <li>Click the <code>&gt;</code> symbol to view the instance list of the model.</li> </ol>"},{"location":"user-guide/model-management/#delete-model-instance","title":"Delete Model Instance","text":"<ol> <li>Find the model you want to check on the model list page.</li> <li>Click the <code>&gt;</code> symbol to view the instance list of the model.</li> <li>Find the model instance you want to delete.</li> <li>Click the ellipsis button for the model instance in the <code>Operations</code> column, then select <code>Delete</code>.</li> <li>Confirm the deletion.</li> </ol> <p>Note</p> <p>After a model instance is deleted, GPUStack will recreate a new instance to satisfy the expected replicas of the model if necessary.</p>"},{"location":"user-guide/model-management/#view-model-instance-logs","title":"View Model Instance Logs","text":"<ol> <li>Find the model you want to check on the model list page.</li> <li>Click the <code>&gt;</code> symbol to view the instance list of the model.</li> <li>Find the model instance you want to check.</li> <li>Click the <code>View Logs</code> button for the model instance in the <code>Operations</code> column.</li> </ol>"},{"location":"user-guide/model-management/#use-self-hosted-ollama-models","title":"Use Self-hosted Ollama Models","text":"<p>You can deploy self-hosted Ollama models by configuring the <code>--ollama-library-base-url</code> option in the GPUStack server. The <code>Ollama Library</code> URL should point to the base URL of the Ollama model registry. For example, <code>https://registry.mycompany.com</code>.</p> <p>Here is an example workflow to set up a registry, publish a model, and use it in GPUStack:</p> <pre><code># Run a self-hosted OCI registry\ndocker run -d -p 5001:5000 --name registry registry:2\n\n# Push a model to the registry using Ollama\nollama pull llama3\nollama cp llama3 localhost:5001/library/llama3\nollama push localhost:5001/library/llama3 --insecure\n\n# Start GPUStack server with the custom Ollama library URL\ncurl -sfL https://get.gpustack.ai | sh -s - --ollama-library-base-url http://localhost:5001\n</code></pre> <p>That's it! You can now deploy the model <code>llama3</code> from <code>Ollama Library</code> source in GPUStack as usual, but the model will now be fetched from the self-hosted registry.</p>"},{"location":"user-guide/model-management/#advanced-model-configuration","title":"Advanced Model Configuration","text":"<p>GPUStack supports tailored configurations for model deployment.</p>"},{"location":"user-guide/model-management/#schedule-type","title":"Schedule Type","text":""},{"location":"user-guide/model-management/#auto","title":"Auto","text":"<p>GPUStack automatically schedules model instances to appropriate GPUs/Workers based on current resource availability.</p> <ul> <li> <p>Placement Strategy</p> </li> <li> <p>Spread: Make the resources of the entire cluster relatively evenly distributed among all workers. It may produce more resource fragmentation on a single worker.</p> </li> <li> <p>Binpack: Prioritize the overall utilization of cluster resources, reducing resource fragmentation on Workers/GPUs.</p> </li> <li> <p>Worker Selector</p> </li> </ul> <p>When configured, the scheduler will deploy the model instance to the worker containing specified labels.</p> <ol> <li> <p>Navigate to the <code>Resources</code> page and edit the desired worker. Assign custom labels to the worker by adding them in the labels section.</p> </li> <li> <p>Go to the <code>Models</code> page and click on the <code>Deploy Model</code> button. Expand the <code>Advanced</code> section and input the previously assigned worker labels in the <code>Worker Selector</code> configuration. During deployment, the Model Instance will be allocated to the corresponding worker based on these labels.</p> </li> </ol>"},{"location":"user-guide/model-management/#manual","title":"Manual","text":"<p>This schedule type allows users to specify which GPU to deploy the model instance on.</p> <ul> <li>GPU Selector</li> </ul> <p>Select a GPU from the list. The model instance will attempt to deploy to this GPU if resources permit.</p>"},{"location":"user-guide/model-management/#backend","title":"Backend","text":"<p>The inference backend. Currently, GPUStack supports three backends: llama-box, vLLM and vox-box. GPUStack automatically selects the backend based on the model's configuration.</p> <p>For more details, please refer to the Inference Backends section.</p>"},{"location":"user-guide/model-management/#backend-version","title":"Backend Version","text":"<p>Specify a backend version, such as <code>v1.0.0</code>. The version format and availability depend on the selected backend. This option is useful for ensuring compatibility or taking advantage of features introduced in specific backend versions. Refer to the Pinned Backend Versions section for more information.</p>"},{"location":"user-guide/model-management/#backend-parameters","title":"Backend Parameters","text":"<p>Input the parameters for the backend you want to customize when running the model. The parameter should be in the format <code>--parameter=value</code>, <code>--bool-parameter</code> or as separate fields for <code>--parameter</code> and <code>value</code>. For example, use <code>--ctx-size=8192</code> for llama-box.</p> <p>For full list of supported parameters, please refer to the Inference Backends section.</p>"},{"location":"user-guide/model-management/#allow-cpu-offloading","title":"Allow CPU Offloading","text":"<p>Note</p> <p>Available for llama-box backend only.</p> <p>After enabling CPU offloading, GPUStack prioritizes loading as many layers as possible onto the GPU to optimize performance. If GPU resources are limited, some layers will be offloaded to the CPU, with full CPU inference used only when no GPU is available.</p>"},{"location":"user-guide/model-management/#allow-distributed-inference-across-workers","title":"Allow Distributed Inference Across Workers","text":"<p>Note</p> <p>Available for llama-box backend only.</p> <p>Enable distributed inference across multiple workers. The primary Model Instance will communicate with backend instances on one or more other workers, offloading computation tasks to them.</p>"},{"location":"user-guide/openai-compatible-apis/","title":"OpenAI Compatible APIs","text":"<p>GPUStack serves OpenAI-compatible APIs using the <code>/v1-openai</code> path. Most of the APIs also work under the <code>/v1</code> path as an alias, except for the <code>models</code> endpoint, which is reserved for GPUStack management APIs.</p>"},{"location":"user-guide/openai-compatible-apis/#supported-endpoints","title":"Supported Endpoints","text":"<p>The following API endpoints are supported:</p> <ul> <li> List Models</li> <li> Create Completion</li> <li> Create Chat Completion</li> <li> Create Embeddings</li> <li> Create Image</li> <li> Create Image Edit</li> <li> Create Speech</li> <li> Create Transcription</li> </ul>"},{"location":"user-guide/openai-compatible-apis/#usage","title":"Usage","text":"<p>The following are examples using the APIs in different languages:</p>"},{"location":"user-guide/openai-compatible-apis/#curl","title":"curl","text":"<pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1-openai/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n  -d '{\n    \"model\": \"llama3\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a helpful assistant.\"\n      },\n      {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n      }\n    ],\n    \"stream\": true\n  }'\n</code></pre>"},{"location":"user-guide/openai-compatible-apis/#openai-python-api-library","title":"OpenAI Python API library","text":"<pre><code>from openai import OpenAI\n\nclient = OpenAI(base_url=\"http://myserver/v1-openai\", api_key=\"myapikey\")\n\ncompletion = client.chat.completions.create(\n  model=\"llama3\",\n  messages=[\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"Hello!\"}\n  ]\n)\n\nprint(completion.choices[0].message)\n</code></pre>"},{"location":"user-guide/openai-compatible-apis/#openai-node-api-library","title":"OpenAI Node API library","text":"<pre><code>const OpenAI = require(\"openai\");\n\nconst openai = new OpenAI({\n  apiKey: \"myapikey\",\n  baseURL: \"http://myserver/v1-openai\",\n});\n\nasync function main() {\n  const params = {\n    model: \"llama3\",\n    messages: [\n      {\n        role: \"system\",\n        content: \"You are a helpful assistant.\",\n      },\n      {\n        role: \"user\",\n        content: \"Hello!\",\n      },\n    ],\n  };\n  const chatCompletion = await openai.chat.completions.create(params);\n  console.log(chatCompletion.choices[0].message);\n}\nmain();\n</code></pre>"},{"location":"user-guide/pinned-backend-versions/","title":"Pinned Backend Versions","text":"<p>Inference engines in the generative AI domain are evolving rapidly to enhance performance and unlock new capabilities. This constant evolution provides exciting opportunities but also presents challenges for maintaining model compatibility and deployment stability.</p> <p>GPUStack allows you to pin inference backend versions to specific releases, offering a balance between staying up-to-date with the latest advancements and ensuring a reliable runtime environment. This feature is particularly beneficial in the following scenarios:</p> <ul> <li>Leveraging the newest backend features without waiting for a GPUStack update.</li> <li>Locking in a specific backend version to maintain compatibility with existing models.</li> <li>Assigning different backend versions to models with varying requirements.</li> </ul> <p>By pinning backend versions, you gain full control over your inference environment, enabling both flexibility and predictability in deployment.</p>"},{"location":"user-guide/pinned-backend-versions/#automatic-installation-of-pinned-backend-versions","title":"Automatic Installation of Pinned Backend Versions","text":"<p>To simplify deployment, GPUStack supports the automatic installation of pinned backend versions when feasible. The process depends on the type of backend:</p> <ol> <li>Prebuilt Binaries    For backends like <code>llama-box</code>, GPUStack downloads the specified version using the same mechanism as in GPUStack bootstrapping.</li> </ol> <p>Tip</p> <p>You can customize the download source using the <code>--tools-download-base-url</code> configuration option.</p> <ol> <li>Python-based Backends    For backends like <code>vLLM</code> and <code>vox-box</code>, GPUStack uses <code>pipx</code> to install the specified version in an isolated Python environment.</li> </ol> <p>Tip</p> <ul> <li>Ensure that <code>pipx</code> is installed on the worker nodes.</li> <li>If <code>pipx</code> is not in the system PATH, specify its location with the <code>--pipx-path</code> configuration option.</li> </ul> <p>This automation reduces manual intervention, allowing you to focus on deploying and using your models.</p>"},{"location":"user-guide/pinned-backend-versions/#manual-installation-of-pinned-backend-versions","title":"Manual Installation of Pinned Backend Versions","text":"<p>When automatic installation is not feasible or preferred, GPUStack provides a straightforward way to manually install specific versions of inference backends. Follow these steps:</p> <ol> <li>Prepare the Executable    Install the backend executable or link it under the GPUStack bin directory. The default locations are:</li> </ol> <ul> <li>Linux/macOS: <code>/var/lib/gpustack/bin</code></li> <li>Windows: <code>$env:AppData\\gpustack\\bin</code></li> </ul> <p>Tip</p> <p>You can customize the bin directory using the <code>--bin-dir</code> configuration option.</p> <ol> <li>Name the Executable    Ensure the executable is named in the following format:</li> </ol> <ul> <li>Linux/macOS: <code>&lt;backend&gt;_&lt;version&gt;</code></li> <li>Windows: <code>&lt;backend&gt;_&lt;version&gt;.exe</code></li> </ul> <p>For example, the vLLM executable for version v0.6.4 should be named <code>vllm_v0.6.4</code> on Linux.</p> <p>By following these steps, you can maintain full control over the backend installation process, ensuring that the correct version is used for your deployment.</p>"},{"location":"user-guide/rerank-api/","title":"Rerank API","text":"<p>In the context of Retrieval-Augmented Generation (RAG), reranking refers to the process of selecting the most relevant information from retrieved documents or knowledge sources before presenting them to the user or utilizing them for answer generation.</p> <p>GPUStack serves Jina compatible Rerank API using the <code>/v1/rerank</code> path.</p> <p>Note</p> <p>The Rerank API is only available when using the llama-box inference backend.</p>"},{"location":"user-guide/rerank-api/#supported-models","title":"Supported Models","text":"<p>The following models are available for reranking:</p> <ul> <li>bce-reranker-base_v1</li> <li>jina-reranker-v1-turbo-en</li> <li>jina-reranker-v1-tiny-en</li> <li>bge-reranker-v2-m3</li> <li>gte-multilingual-reranker-base \ud83e\uddea</li> <li>jina-reranker-v2-base-multilingual \ud83e\uddea</li> </ul>"},{"location":"user-guide/rerank-api/#usage","title":"Usage","text":"<p>The following is an example using the Rerank API:</p> <pre><code>export GPUSTACK_API_KEY=myapikey\ncurl http://myserver/v1/rerank \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Authorization: Bearer $GPUSTACK_API_KEY\" \\\n    -d '{\n        \"model\": \"bge-reranker-v2-m3\",\n        \"query\": \"What is a panda?\",\n        \"top_n\": 3,\n        \"documents\": [\n            \"hi\",\n            \"it is a bear\",\n            \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n        ]\n    }' | jq\n</code></pre> <p>Example output:</p> <pre><code>{\n  \"model\": \"bge-reranker-v2-m3\",\n  \"object\": \"list\",\n  \"results\": [\n    {\n      \"document\": {\n        \"text\": \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\"\n      },\n      \"index\": 2,\n      \"relevance_score\": 1.951932668685913\n    },\n    {\n      \"document\": {\n        \"text\": \"it is a bear\"\n      },\n      \"index\": 1,\n      \"relevance_score\": -3.7347371578216553\n    },\n    {\n      \"document\": {\n        \"text\": \"hi\"\n      },\n      \"index\": 0,\n      \"relevance_score\": -6.157620906829834\n    }\n  ],\n  \"usage\": {\n    \"prompt_tokens\": 69,\n    \"total_tokens\": 69\n  }\n}\n</code></pre>"},{"location":"user-guide/user-management/","title":"User Management","text":"<p>GPUStack supports users of two roles: <code>Admin</code> and <code>User</code>. Admins can monitor system status, manage models, users, and system settings. Users can manage their own API keys and use the completion API.</p>"},{"location":"user-guide/user-management/#default-admin","title":"Default Admin","text":"<p>On bootstrap, GPUStack creates a default admin user. The initial password for the default admin is stored in <code>&lt;data-dir&gt;/initial_admin_password</code>. In the default setup, it should be <code>/var/lib/gpustack/initial_admin_password</code>. You can customize the default admin password by setting the <code>--bootstrap-password</code> parameter when starting <code>gpustack</code>.</p>"},{"location":"user-guide/user-management/#create-user","title":"Create User","text":"<ol> <li>Navigate to the <code>Users</code> page.</li> <li>Click the <code>Create User</code> button.</li> <li>Fill in <code>Name</code>, <code>Full Name</code>, <code>Password</code>, and select <code>Role</code> for the user.</li> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"user-guide/user-management/#update-user","title":"Update User","text":"<ol> <li>Navigate to the <code>Users</code> page.</li> <li>Find the user you want to edit.</li> <li>Click the <code>Edit</code> button in the <code>Operations</code> column.</li> <li>Update the attributes as needed.</li> <li>Click the <code>Save</code> button.</li> </ol>"},{"location":"user-guide/user-management/#delete-user","title":"Delete User","text":"<ol> <li>Navigate to the <code>Users</code> page.</li> <li>Find the user you want to delete.</li> <li>Click the ellipsis button in the <code>Operations</code> column, then select <code>Delete</code>.</li> <li>Confirm the deletion.</li> </ol>"},{"location":"user-guide/playground/","title":"Playground","text":"<p>GPUStack offers a playground UI where users can test and experiment with the APIs. Refer to each subpage for detailed instructions and information.</p>"},{"location":"user-guide/playground/audio/","title":"Audio Playground","text":"<p>The Audio Playground is a dedicated space for testing and experimenting with GPUStack\u2019s text-to-speech (TTS) and speech-to-text (STT) APIs. It allows users to interactively convert text to audio and audio to text, customize parameters, and review code examples for seamless API integration.</p>"},{"location":"user-guide/playground/audio/#text-to-speech","title":"Text to Speech","text":"<p>Switch to the \"Text to Speech\" tab to test TTS models.</p>"},{"location":"user-guide/playground/audio/#text-input","title":"Text Input","text":"<p>Enter the text you want to convert, then click the <code>Submit</code> button to generate the corresponding speech.</p> <p></p>"},{"location":"user-guide/playground/audio/#clear-text","title":"Clear Text","text":"<p>Click the <code>Clear</code> button to reset the text input and remove the generated speech.</p>"},{"location":"user-guide/playground/audio/#select-model","title":"Select Model","text":"<p>Select an available TTS model in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/audio/#customize-parameters","title":"Customize Parameters","text":"<p>Customize the voice and format of the audio output.</p> <p>Tip</p> <p>Supported voices may vary between models.</p>"},{"location":"user-guide/playground/audio/#view-code","title":"View Code","text":"<p>After experimenting with input text and parameters, click the <code>View Code</code> button to see how to call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/audio/#speech-to-text","title":"Speech to Text","text":"<p>Switch to the \"Speech to Text\" tab to test STT models.</p>"},{"location":"user-guide/playground/audio/#provide-audio-file","title":"Provide Audio File","text":"<p>You can provide audio for transcription in two ways:</p> <ol> <li>Upload an audio file.</li> <li>Record audio online.</li> </ol> <p>Note</p> <p>If the online recording is not available, it could be due to one of the following reasons:</p> <ol> <li>For HTTPS or <code>http://localhost</code> access, microphone permissions must be enabled in your browser.</li> <li> <p>For access via <code>http://{host IP}</code>, the URL must be added to your browser's trusted list.</p> <p>Example:   In Chrome, navigate to <code>chrome://flags/</code>, add the GPUStack URL to \"Insecure origins treated as secure,\" and enable this option.</p> </li> </ol> <p></p> <p></p>"},{"location":"user-guide/playground/audio/#select-model_1","title":"Select Model","text":"<p>Select an available STT model in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/audio/#copy-text","title":"Copy Text","text":"<p>Copy the transcription results generated by the model.</p>"},{"location":"user-guide/playground/audio/#customize-parameters_1","title":"Customize Parameters","text":"<p>Select the appropriate language for your audio file to optimize transcription accuracy.</p>"},{"location":"user-guide/playground/audio/#view-code_1","title":"View Code","text":"<p>After experimenting with audio files and parameters, click the <code>View Code</code> button to see how to call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/chat/","title":"Chat Playground","text":"<p>Interact with the chat completions API. The following is an example screenshot:</p> <p></p>"},{"location":"user-guide/playground/chat/#prompts","title":"Prompts","text":"<p>You can adjust the prompt messages on the left side of the playground. There are three role types of prompt messages: system, user, and assistant.</p> <ul> <li>System: Typically a predefined instruction or guidance that sets the context, defines the behavior, or imposes specific constraints on how the model should generate its responses.</li> <li>User: The input or query provided by the user (the person interacting with the LLM).</li> <li>Assistant: The response generated by the LLM.</li> </ul>"},{"location":"user-guide/playground/chat/#edit-system-message","title":"Edit System Message","text":"<p>You can add and edit the system message at the top of the playground.</p>"},{"location":"user-guide/playground/chat/#edit-user-and-assistant-messages","title":"Edit User and Assistant Messages","text":"<p>To add a user or assistant message, click the <code>New Message</code> button.</p> <p>To remove a user or assistant message, click the minus button at the right corner of the message.</p> <p>To change the role of a message, click the <code>User</code> or <code>Assistant</code> text at the beginning of the message.</p>"},{"location":"user-guide/playground/chat/#upload-image","title":"Upload Image","text":"<p>You can add images to the prompt by clicking the <code>Upload Image</code> button.</p>"},{"location":"user-guide/playground/chat/#clear-prompts","title":"Clear Prompts","text":"<p>Click the <code>Clear</code> button to clear all the prompts.</p>"},{"location":"user-guide/playground/chat/#select-model","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-right corner of the playground. Please refer to Model Management to learn about how to manage models.</p>"},{"location":"user-guide/playground/chat/#customize-parameters","title":"Customize Parameters","text":"<p>You can customize completion parameters in the <code>Parameters</code> section.</p>"},{"location":"user-guide/playground/chat/#do-completion","title":"Do Completion","text":"<p>You can do a completion by clicking the <code>Submit</code> button.</p>"},{"location":"user-guide/playground/chat/#view-code","title":"View Code","text":"<p>Once you've done experimenting with the prompts and parameters, you can click the <code>View Code</code> button to check how you can call the API with the same input by code. Code examples in <code>curl</code>, <code>Python</code>, and <code>Node.js</code> are provided.</p>"},{"location":"user-guide/playground/chat/#compare-playground","title":"Compare Playground","text":"<p>You can compare multiple models in the playground. The following is an example screenshot:</p> <p></p>"},{"location":"user-guide/playground/chat/#comparision-mode","title":"Comparision Mode","text":"<p>You can choose the number of models to compare by clicking the comparison view buttons, including 2, 3, 4 and 6-model comparison.</p>"},{"location":"user-guide/playground/chat/#prompts_1","title":"Prompts","text":"<p>You can adjust the prompt messages similar to the chat playground.</p>"},{"location":"user-guide/playground/chat/#upload-image_1","title":"Upload Image","text":"<p>You can add images to the prompt by clicking the <code>Upload Image</code> button.</p>"},{"location":"user-guide/playground/chat/#clear-prompts_1","title":"Clear Prompts","text":"<p>Click the <code>Clear</code> button to clear all the prompts.</p>"},{"location":"user-guide/playground/chat/#select-model_1","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-left corner of each model panel.</p>"},{"location":"user-guide/playground/chat/#customize-parameters_1","title":"Customize Parameters","text":"<p>You can customize completion parameters by clicking the settings button of each model.</p>"},{"location":"user-guide/playground/embedding/","title":"Embedding Playground","text":"<p>The Embedding Playground lets you test the model\u2019s ability to convert text into embeddings. It allows you to experiment with multiple text inputs, visualize embeddings, and review code examples for API integration.</p>"},{"location":"user-guide/playground/embedding/#add-text","title":"Add Text","text":"<p>Add at least two text entries and click the <code>Submit</code> button to generate embeddings.</p>"},{"location":"user-guide/playground/embedding/#batch-input-text","title":"Batch Input Text","text":"<p>Enable <code>Batch Input Mode</code> to automatically split multi-line text into separate entries based on line breaks. This is useful for processing multiple text snippets in a single operation.</p>"},{"location":"user-guide/playground/embedding/#visualization","title":"Visualization","text":"<p>Visualize the embedding results using PCA (Principal Component Analysis) to reduce dimensions and display them on a 2D plot. Results can be viewed in two formats:</p> <ol> <li>Chart - Display PCA results visually.</li> <li>JSON - View raw embeddings in JSON format.</li> </ol> <p>In the chart, the distance between points represents the similarity between corresponding texts. Closer points indicate higher similarity.</p> <p></p>"},{"location":"user-guide/playground/embedding/#clear","title":"Clear","text":"<p>Click the <code>Clear</code> button to reset text entries and clear the output.</p>"},{"location":"user-guide/playground/embedding/#select-model","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/embedding/#view-code","title":"View Code","text":"<p>After experimenting with the text inputs, click the <code>View Code</code> button to see how you can call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/image/","title":"Image Playground","text":"<p>The Image Playground is a dedicated space for testing and experimenting with GPUStack\u2019s image generation APIs. It allows users to interactively explore the capabilities of different models, customize parameters, and review code examples for seamless API integration.</p>"},{"location":"user-guide/playground/image/#prompt","title":"Prompt","text":"<p>You can input or randomly generate a prompt, then click the Submit button to generate an image.</p> <p></p>"},{"location":"user-guide/playground/image/#clear-prompt","title":"Clear Prompt","text":"<p>Click the <code>Clear</code> button to reset the prompt and remove the generated image.</p>"},{"location":"user-guide/playground/image/#select-model","title":"Select Model","text":"<p>You can select available models in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/image/#customize-parameters","title":"Customize Parameters","text":"<p>You can customize the image generation parameters by switching between two API styles:</p> <ol> <li>OpenAI-compatible mode.</li> <li>Advanced mode.</li> </ol> <p></p>"},{"location":"user-guide/playground/image/#advanced-parameters","title":"Advanced Parameters","text":"Parameter Default Description <code>Counts</code> <code>1</code> Number of images to generate. <code>Size</code> <code>512x512</code> The size of the generated image in 'widthxheight' format. <code>Sampler</code> <code>euler_a</code> The sampler algorithm for image generation. Options include 'euler_a', 'euler', 'heun', 'dpm2', 'dpm++2s_a', 'dpm++2m', 'dpm++2mv2', 'ipndm', 'ipndm_v', and 'lcm'. <code>Schedule</code> <code>discrete</code> The noise scheduling method. <code>Sampler Steps</code> <code>10</code> The number of sampling steps to perform. Higher values may improve image quality at the cost of longer processing time. <code>CFG Scale</code> <code>4.5</code> The scale for classifier-free guidance. A higher value increases adherence to the prompt. <code>Negative Prompt</code> (empty) A negative prompt to specify what the image should avoid. <code>Seed</code> (empty) Random seed. <p>Note</p> <p>The maximum image size is restricted by the model's deployment settings. See the diagram below:</p> <p></p>"},{"location":"user-guide/playground/image/#view-code","title":"View Code","text":"<p>After experimenting with prompts and parameters, click the <code>View Code</code> button to see how to call the API with the same inputs. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"},{"location":"user-guide/playground/rerank/","title":"Rerank Playground","text":"<p>The Rerank Playground allows you to test reranker models that reorder multiple texts based on their relevance to a query. Experiment with various input texts, customize parameters, and review code examples for API integration.</p>"},{"location":"user-guide/playground/rerank/#add-text","title":"Add Text","text":"<p>Add multiple text entries to the document for reranking.</p>"},{"location":"user-guide/playground/rerank/#bach-input-text","title":"Bach Input Text","text":"<p>Enable <code>Batch Input Mode</code> to split multi-line text into separate entries based on line breaks. This is useful for processing multiple text snippets efficiently.</p>"},{"location":"user-guide/playground/rerank/#clear","title":"Clear","text":"<p>Click the <code>Clear</code> button to reset the document and query results.</p>"},{"location":"user-guide/playground/rerank/#query","title":"Query","text":"<p>Input a query and click the <code>Submit</code> button to get a ranked list of texts based on their relevance to the query.</p> <p></p>"},{"location":"user-guide/playground/rerank/#select-model","title":"Select Model","text":"<p>Select an available reranker model in GPUStack by clicking the model dropdown at the top-right corner of the playground UI.</p>"},{"location":"user-guide/playground/rerank/#customize-parameters","title":"Customize Parameters","text":"<p>In the parameter section, set <code>Top N</code> to specify the number of matching texts to retrieve.</p>"},{"location":"user-guide/playground/rerank/#view-code","title":"View Code","text":"<p>After experimenting with the input text and query, click the <code>View Code</code> button to see how to call the API with the same input. Code examples are provided in <code>curl</code>, <code>Python</code>, and <code>Node.js</code>.</p>"}]}
\ No newline at end of file
diff --git a/0.4/sitemap.xml b/0.4/sitemap.xml
index 65a3268..6231bbb 100644
--- a/0.4/sitemap.xml
+++ b/0.4/sitemap.xml
@@ -2,237 +2,237 @@
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <url>
          <loc>https://docs.gpustack.ai/0.4/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/api-reference/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/architecture/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/code-of-conduct/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/contributing/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/development/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/overview/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/quickstart/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/scheduler/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/troubleshooting/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/upgrade/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/cli-reference/chat/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/cli-reference/download-tools/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/cli-reference/draw/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/cli-reference/start/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/installation/air-gapped-installation/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/installation/docker-installation/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/installation/installation-requirements/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/installation/installation-script/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/installation/manual-installation/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/installation/uninstallation/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/creating-text-embeddings/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/inference-on-cpus/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/inference-with-function-calling/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/performing-distributed-inference-across-workers/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/running-inference-with-ascend-npus/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/running-inference-with-moorethreads-gpus/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/setting-up-a-multi-node-gpustack-cluster/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/using-audio-models/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/using-image-generation-models/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/using-reranker-models/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/tutorials/using-vision-language-models/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/api-key-management/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/image-generation-apis/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/inference-backends/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/model-management/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/openai-compatible-apis/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/pinned-backend-versions/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/rerank-api/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/user-management/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/playground/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/playground/audio/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/playground/chat/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/playground/embedding/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/playground/image/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
     <url>
          <loc>https://docs.gpustack.ai/0.4/user-guide/playground/rerank/</loc>
-         <lastmod>2024-12-13</lastmod>
+         <lastmod>2024-12-17</lastmod>
          <changefreq>daily</changefreq>
     </url>
 </urlset>
\ No newline at end of file
diff --git a/0.4/sitemap.xml.gz b/0.4/sitemap.xml.gz
index e23da6c03059ce3152f9d4e4b1f5d83002b19dc6..32fd4318f66eaea52f464ec3b413dc9a82bb1304 100644
GIT binary patch
delta 696
zcmV;p0!RJG1;_;lABzYGfKFhM2OfW?Gy5?~dg~MHJ^%s5W`YrwAa37&g`Kw9ufra8
z27F1($mA#ak0i+U@n!U@GY~rryY=1XcD;gN(7Mp?*5AK9sn6@j{mnL$XGkkC4!m0r
zDaEf{H%-%qU4l_Z-XQfIn?Y{4y9{=>n=kAA%}TB^!SZdswjRyC%)&B_MZ<raK4wnZ
zoHp9YV-MZ7D<;=ZXig*A{r&CzgSxv{cb~-MGInKV1|51h5<Ks%cK&tSU5zX#-8;)a
zQ|agw!4XJe1hq1f86^HDWvUfX^aTo!%3!eBq-rTcB6rA1)<UI-7S7-?ju2`MB1&#K
zJD4iPd(O_BWQ1ysUrDJpd!T<x`4VA1c;ErC;&qZ^Puf<S4*hjik(pBypcbd#QCpP|
zJ-2~OdTIi+=9J`u+<LE*w2Ym0q<S4ASoPC*1L$IL0%{KX`bHEZ6^x)X0Nflr8*;G;
zm5(}P?VFQI4h;~eoRFNB3StO!X(CjDm!x1kfGzt_t{&WlOWszZY9oJU4G~W+4eFSK
znU|yyvDDkfeeg%1BhpyJVjVBrw7CSmxup#uvzm}3$JhTpD2Gs-`Q^Q6HV~`|ElBDw
zE{uo-X^^FDS@kVBQkDZo=P{}1GplHLafySD(n<DW@KU`u3~A1(IV!Ej?2}WW9KXCh
zp^W&J>fK+yJCeD$p|gK=D5A39xe;VH>R#dv0ch!anxZG64y8|Siu&xzfwuDU3%NwU
zYWU0&-ab1E^8xb-UX?VQf?e^s-WBt2`W7b|2=zyuI*5Uk*#?+r88;;kj1aVwLOMnf
z$H7A*>WD4`X(%t!P?vwm8USDNb%}EQtn=5F)EvEj?Fn<RO|dA>O;c0E6(55C8TkFQ
eqABXRbT>u3q__W%Jb%dEY5xQGIjv$MApiihBvRr4

delta 696
zcmV;p0!RJG1;_;lABzYGfCyWW2OfX7huxhdz4ZxpAAo>jGr@>T5Vvo?!cN=l*I^Gk
z1HL3?Wb%{zM-pWF_%iy{8HgQ*-TH2GyIw&sXkF-c>+j#5)aUi%{$`uWGo+Om2i~oR
zl;YQ}o2F^QF2SfHZ;<+q%^)}2T?V_`&6oB5W+hjdVEHy*TaRX6W?`AeqTzo{A2TOy
zP8;pyv4?Kk6_e{HG^Y{m{{Hs<LEYV}yH8?r8M`twgAP3$37+>>JO8@vu0|G=?w#eI
zsdRLT;0Po!f?65L3=)5nGS!MG`T~VVWiZ%mQni#JkvrrhYoSs^3uo{cM+mhB5hb^r
z9ZZ$tJ!fZ5GD5Y+ucTC)Jy3t8e2Fk0Jn(>6@jA(|CvB@uhyJ>%$jqq;P>WOWsI5wf
zp4&hsJv9MZb4qeSZoSt@TE<R0QoW85tomuZ0d%oA0X2txeItsI3Pw;G0B#PR4Y}Ba
z%10ft_RUEphX#mKPDoBm1u+D=G!ZJnOHwc%z?OX|R}b#OC2uQHwGn@_hKMJZ26fEA
z%uCXUSn6%#KKLWh5os)9v5uE*+FXL(+|q`SSxrci<LiGPltU=a{PJEj8wgf~79{l-
z7e+*aG|1AntooK5Da!$)^O#ihnN>8rxWqw6=_GqGc&XkShBW8Y9F<mM_Q|PGj$huM
zP)2-9_3kg<9m!nW(Aj@F6j535+z7H8buaOT0JQWyP0^E3htelEMSXVVKwEkFg<PUv
zHGF0XZ=aoo`GENZuSyzD!LImR?~3_1eTx$fg!-dS9mGJ&Yy-@*jGGb%MhMzTAswTL
z<KUqYbwn3}G?W)<sLMZO4S+BCx<t8t*7<8oYK~sN_JldurdTNFrl~37iVwm64E%mt
e(G>Mux|<?i(%b(>o<C&owEqE!MW@^%Apiiv>`?;%

diff --git a/0.4/tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/index.html b/0.4/tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/index.html
index 4fb1e20..4eec4eb 100644
--- a/0.4/tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/index.html
+++ b/0.4/tutorials/running-on-copilot-plus-pcs-with-snapdragon-x/index.html
@@ -1635,7 +1635,7 @@ <h1 id="running-inference-on-copilot-pcs-with-snapdragon-x">Running Inference on
 <h2 id="prerequisites">Prerequisites</h2>
 <ul>
 <li>A Copilot+ PC with Snapdragon X. In this tutorial, we use the Dell XPS 13 9345.</li>
-<li>Install <strong>AMD64</strong> Python 3.10 or above. <a href="../../installation/installation-requirements/#supported-architectures">See details</a></li>
+<li>Install <strong>AMD64</strong> Python (version 3.10 to 3.12). <a href="../../installation/installation-requirements/#supported-architectures">See details</a></li>
 </ul>
 <h2 id="installing-gpustack">Installing GPUStack</h2>
 <p>Run PowerShell as administrator (avoid using PowerShell ISE), then run the following command to install GPUStack:</p>