Skip to content

Commit

Permalink
Add information for on-demand model switching in Bedrock (#1648)
Browse files Browse the repository at this point in the history
  • Loading branch information
silv-io authored Feb 6, 2025
1 parent 56c4346 commit 929076f
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 1 deletion.
3 changes: 2 additions & 1 deletion content/en/references/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,8 @@ This section covers configuration options that are specific to certain AWS servi
| Variable | Example Values | Description |
| - | - | - |
| `BEDROCK_PREWARM` | `0` (default) \| `1` | Pre-warm the Bedrock engine directly on LocalStack startup instead of on demand. |
| `DEFAULT_BEDROCK_MODEL` | `smollm2:360m` (default) | The model to use to handle text model invocations in Bedrock. Any text-based model available for Ollama is usable. |
| `DEFAULT_BEDROCK_MODEL` | `smollm2:360m` (default) | The model that is used initially to handle text model invocations in Bedrock. Any text-based model available for Ollama is usable. |
| `BEDROCK_PULL_MODELS` | `deepseek-r1,mistral` \' '' (default) | A list of models that should get pulled into the model cache on startup. `DEFAULT_BEDROCK_MODEL` is automatically in there |

### BigData (EMR, Athena, Glue)

Expand Down
18 changes: 18 additions & 0 deletions content/en/user-guide/aws/bedrock/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ The startup of the Bedrock engine can take some time.
Per default, we only start it once you send a request to one of the `bedrock-runtime` APIs.
However, if you want to start the engine when localstack starts to avoid long wait times on your first request you can set the flag `BEDROCK_PREWARM`.

On startup, the `DEFAULT_BEDROCK_MODEL` is pulled from the Ollama library and loaded into memory.
However, you can define an additional list of models in `BEDROCK_PULL_MODELS` to pull additional models when the Bedrock engine starts up.
This way you avoid long wait times when switching between models on demand with requests.

### List available foundation models

You can view all available foundation models using the [`ListFoundationModels`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListFoundationModels.html) API.
Expand Down Expand Up @@ -140,6 +144,20 @@ For example, to use the Mistral model, set the environment variable while starti
$ DEFAULT_BEDROCK_MODEL=mistral localstack start
{{< / command >}}

You can also define models directly in the request, by setting the `model-id` parameter to `ollama.<ollama-model-id>`.
For example, if you want to access `deepseek-r1`, you can do it like this:

{{< command >}}
$ awslocal bedrock-runtime converse \
--model-id "ollama.deepseek-r1" \
--messages '[{
"role": "user",
"content": [{
"text": "Say Hello!"
}]
}]'
{{< / command >}}

## Troubleshooting

Users of Docker Desktop on macOS or Windows might run into the issue of Bedrock becoming unresponsive after some usage.
Expand Down

0 comments on commit 929076f

Please sign in to comment.