+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 +10 +11 +12 +13 +14 +15 +16 +17 +18 +19 +20 +21 +22 +23 +24 +25 +26 +27 +28 +29 +30 +31 +32 +33 +34 +35 +36 +37 +38 +39 +40 +41 +42 +43 +44 +45 | |
1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 +10 +11 +12 +13 +14 +15 +16 +17 +18 +19 +20 +21 +22 +23 +24 +25 +26 +27 +28 +29 +30 +31 +32 +33 +34 +35 +36 +37 +38 +39 +40 +41 +42 +43 +44 +45 +46 +47 +48 +49 +50 +51 +52 +53 +54 +55 +56 +57 +58 +59 +60 +61 +62 +63 +64 +65 +66 +67 +68 +69 +70 +71 +72 +73 +74 +75 +76 | |
Copy the code from llm_guard_api
+Install dependencies (preferably in a virtual environment) +
python -m pip install ".[cpu]"
+python -m pip install ".[gpu]" # If you have a GPU
+Alternatively, you can use Makefile: +
make install
+Run the API locally:
+make run
+Or using CLI:
+llm_guard_api ./config/scanners.yml
+In case you want to use gunicorn to run the API, you can use the following command:
gunicorn --workers 1 --preload --worker-class uvicorn.workers.UvicornWorker 'app.app:create_app(config_file="./config/scanners.yml")'
+It will preload models in the shared memory among workers, which can be useful for performance.
+Either build the Docker image or pull our official image from Docker Hub.
+In order to build the Docker image, run the following command:
+make build-docker-multi
+make build-docker-cuda-multi # If you have a GPU
+Or pull the official image:
+docker pull laiyer/llm-guard-api:latest
+Now, you can run the Docker container:
+docker run -d -p 8000:8000 -e LOG_LEVEL='DEBUG' -e AUTH_TOKEN='my-token' laiyer/llm-guard-api:latest
+This will start the API on port 8000. You can now access the API at http://localhost:8000/swagger.json.
If you want to use a custom configuration, you can mount a volume to /home/user/app/config:
docker run -d -p 8000:8000 -e LOG_LEVEL='INFO' -v ./config:/home/user/app/config laiyer/llm-guard-api:latest
+Warning
+We recommend at least 16GB of RAM allocated to Docker. We are working on optimizing the memory usage when the container starts.
+If you get an out-of-memory error, you can change config.yml file to use less scanners.
+Alternatively, you can enable low_cpu_mem_usage in scanners that rely on HuggingFace models.
If you get a failed HTTP probe, it might be because the API is still starting. You can increase the initialDelaySeconds in the Kubernetes deployment.
Alternatively, you can configure lazy_load in the YAML config file to load models only on the first request.
LLM Guard can be deployed as an API. We rely on FastAPI and Uvicorn to serve the API.
+All configurations are stored in config/scanners.yml. It supports configuring via environment variables.
Note
+Scanners will be executed in the order of configuration.
+LOG_LEVEL (bool): Log level. Default is INFO. If set as DEBUG, debug mode will be enabled, which makes Swagger UI available.CACHE_MAX_SIZE (int): Maximum number of items in the cache. Default is unlimited.CACHE_TTL (int): Time in seconds after which a cached item expires. Default is 1 hour.SCAN_FAIL_FAST (bool): Stop scanning after the first failed check. Default is False.SCAN_PROMPT_TIMEOUT (int): Time in seconds after which a prompt scan will timeout. Default is 10 seconds.SCAN_OUTPUT_TIMEOUT (int): Time in seconds after which an output scan will timeout. Default is 30 seconds.APP_PORT (int): Port to run the API. Default is 8000.SCAN_FAIL_FAST to avoid unnecessary scans.CACHE_MAX_SIZE and CACHE_TTL to cache results and avoid unnecessary scans.It's possible to load models from a local directory.
+You can set model_path in each supported scanner with the folder to the ONNX version of the model.
This way, the models won't be downloaded each time the container starts.
+ +You can enable lazy_load in the YAML config file to load models only on the first request instead of the API start.
+That way, you can avoid failed HTTP probes due to the long model loading time.
There are built-in environment variables to configure observability:
+ +Logs are written to stdout in a structured format, which can be easily parsed by log management systems.
The following exporters are available for metrics:
+stdout./metrics endpoint.The following exporters are available for tracing:
+stdout