diff --git a/adr/ADR-8.md b/adr/ADR-8.md index 61004479..0ef3976a 100644 --- a/adr/ADR-8.md +++ b/adr/ADR-8.md @@ -1,18 +1,24 @@ # JetStream based Key-Value Stores -|Metadata|Value| -|--------|-----| -|Date |2021-06-30| -|Author |@ripienaar| -|Status |Implemented| -|Tags |jetstream, client, kv| +| Metadata | Value | +|----------|-----------------------| +| Date | 2021-06-30 | +| Author | @ripienaar | +| Status | Implemented | +| Tags | jetstream, client, kv | -## Context +## Release History + +| Revision | Date | Description | +|----------|------------|-------------------------------------------| +| 1 | 2023-10-16 | Document NATS Server 2.10 sourced buckets | +| 1 | 2023-10-16 | Document read replica mirrors buckets | -This document describes a design and initial implementation of a JetStream backed key-value store. The initial implementation -is available in the CLI as `nats kv` with the reference client implementation being the `nats.go` repository. -This document aims to guide client developers in implementing this feature in the language clients we maintain. +## Context + +This document describes a design and initial implementation of a JetStream backed key-value store. All tier-1 clients +support KV. ## Status and Roadmap @@ -41,16 +47,17 @@ additional behaviors will come during the 1.x cycle. ### 1.1 - * Encoders and Decoders for keys and values - * Additional Operation that indicates server limits management deleted messages + * Merged buckets using NATS Server 2.10 subject transforms + * Read replicas facilitated by Stream Mirrors ### 1.2 - * Read replicas facilitated by Stream Mirrors * Read-only operation mode * Replica auto discovery * Read cache against with replica support * Ranged operations + * Encoders and Decoders for keys and values + * Additional Operation that indicates server limits management deleted messages ### 1.3 @@ -297,11 +304,18 @@ Deleted data - (see later section on deletes) - has the `KV-Operation` header se into a `key not found` error in basic gets and into a `Entry` with the correct operation value set in watchers or history. ##### Get Operation + +###### When allow_direct is false We have extended the `io.nats.jetstream.api.v1.stream_msg_get_request` API to support loading the latest value for a specific -subject. Thus a read for `CONFIGURATION.username` becomes a `io.nats.jetstream.api.v1.stream_msg_get_request` with the +subject. Thus, a read for `CONFIGURATION.username` becomes a `io.nats.jetstream.api.v1.stream_msg_get_request` with the `last_by_subj` set to `$KV.CONFIGURATION.auth.username`. +###### When allow_direct is true + +We have introduced a new direct API that allows retrieving the last message for a subject via `$JS.APIDIRECT.GET. +.`. This should be used for performing all gets on a bucket if direct is enabled. + ##### History These operations require access to all values for a key, to achieve this we create an ephemeral consumer on filtered by the subject @@ -360,15 +374,83 @@ first time any message has a `Pending==0`. This signal must also be sent if no d Whatchers should support at least the following options. Languages can choose to support more models if they wish, as long as that is clearly indicated as a language specific extension. Names should be language idiomatic but close to these below. -|Name|Description| -|----|-----------| -|`IncludeHistory`|Send all available history rather than just the latest entries| -|`IgnoreDeletes`|Only sends `PUT` operation entries| -|`MetaOnly`|Does not send any values, only metadata about those values| -|`UpdatesOnly`|Only sends new updates made, no current or historical values are sent. The End Of Initial Data marker is sent as soon as the watch starts.| +| Name | Description | +|------------------|--------------------------------------------------------------------------------------------------------------------------------------------| +| `IncludeHistory` | Send all available history rather than just the latest entries | +| `IgnoreDeletes` | Only sends `PUT` operation entries | +| `MetaOnly` | Does not send any values, only metadata about those values | +| `UpdatesOnly` | Only sends new updates made, no current or historical values are sent. The End Of Initial Data marker is sent as soon as the watch starts. | The default behavior with no options set is to send all the `last_per_subject` values, including delete/purge operations. +#### Replicas for NATS Server 2.10 and newer + +Since NATS Server 2.10 we support transforming messages as a stream configuration item. This allows us to source one +bucket from another and rewrite the keys in the new bucket to have the correct name. + +To copy the keys `NEW.>` from bucket `ORDERS` into `NEW_ORDERS` we create the new stream with the following +partial config: + +```json + "sources": [ + { + "name": "KV_JACO", + "filter_subject": "$KV.ORDERS.NEW.>", + "subject_transforms": [ + { + "src": "$KV.ORDERS.>", + "dest": "$KV.NEW_ORDERS.>" + } + ] + } + ], +``` + +This results in all messages from `ORDERS` keys `NEW.>` to be copied into `NEW_ORDERS` and the subjects rewritten on +write to the new bucket so that a unmodified KV client on `NEW_ORDERS` would just work. + +As this is a `Source` and not a `Mirror` this new bucket can accept writes, we should prioritise a read-only capability +to safeguard against accidental data problems. + +#### Read replica mirrors + +Regional read replicas can be built using the standard mirror feature by setting `mirror_direct` to true as long as the +origin bucket also has `allow_direct`. + +When a direct read is done the response will be from the rtt-nearest mirror. With a mirror added the `nats` command +can be used to verify that a alternative location is set: + +``` +$ nats s info KV_ORDERS +... +State: + + Alternates: KV_ORDERS_NYC: Cluster: nyc Domain: hub + KV_ORDERS: Cluster: lon Domain: hub + +``` + +Here we see a RTT-sorted list of alternatives, the `KV_ORDERS_MIRROR` is nearest to me in the RTT sorted list. + +When doing a direct get the headers will confirm the mirror served the request: + +``` +$ nats req '$JS.API.DIRECT.GET.KV_ORDERS.$KV.ORDERS.NEW.123' '' +13:26:06 Sending request on "JS.API.DIRECT.GET.KV_ORDERS.$KV.ORDERS.NEW.123" +13:26:06 Received with rtt 1.319085ms +13:26:06 Nats-Stream: KV_ORDERS_NYC +13:26:06 Nats-Subject: $KV.ORDERS.NEW.123 +13:26:06 Nats-Sequence: 12 +13:26:06 Nats-Time-Stamp: 2023-10-16T12:54:19.409051084Z + +{......} +``` + +As mirrors support subject filters these regional replicas can hold region specific keys. + +As this is a `Mirror` this stream does not listen on a subject and so the only way to get data into it is via the origin +bucket. + #### API Design notes The API here represents a minimum, languages can add local flavour to the API - for example one can add `PutUint64()` and `GetUint64()`