Skip to content

Commit

Permalink
Document using 2.10 sources to create replicas of buckets
Browse files Browse the repository at this point in the history
Signed-off-by: R.I.Pienaar <[email protected]>
  • Loading branch information
ripienaar committed Oct 16, 2023
1 parent 707da31 commit 857bbd0
Showing 1 changed file with 102 additions and 20 deletions.
122 changes: 102 additions & 20 deletions adr/ADR-8.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
# JetStream based Key-Value Stores

|Metadata|Value|
|--------|-----|
|Date |2021-06-30|
|Author |@ripienaar|
|Status |Implemented|
|Tags |jetstream, client, kv|
| Metadata | Value |
|----------|-----------------------|
| Date | 2021-06-30 |
| Author | @ripienaar |
| Status | Implemented |
| Tags | jetstream, client, kv |

## Context
## Release History

| Revision | Date | Description |
|----------|------------|-------------------------------------------|
| 1 | 2023-10-16 | Document NATS Server 2.10 sourced buckets |
| 1 | 2023-10-16 | Document read replica mirrors buckets |

This document describes a design and initial implementation of a JetStream backed key-value store. The initial implementation
is available in the CLI as `nats kv` with the reference client implementation being the `nats.go` repository.

This document aims to guide client developers in implementing this feature in the language clients we maintain.
## Context

This document describes a design and initial implementation of a JetStream backed key-value store. All tier-1 clients
support KV.

## Status and Roadmap

Expand Down Expand Up @@ -41,16 +47,17 @@ additional behaviors will come during the 1.x cycle.

### 1.1

* Encoders and Decoders for keys and values
* Additional Operation that indicates server limits management deleted messages
* Merged buckets using NATS Server 2.10 subject transforms
* Read replicas facilitated by Stream Mirrors

### 1.2

* Read replicas facilitated by Stream Mirrors
* Read-only operation mode
* Replica auto discovery
* Read cache against with replica support
* Ranged operations
* Encoders and Decoders for keys and values
* Additional Operation that indicates server limits management deleted messages

### 1.3

Expand Down Expand Up @@ -297,11 +304,18 @@ Deleted data - (see later section on deletes) - has the `KV-Operation` header se
into a `key not found` error in basic gets and into a `Entry` with the correct operation value set in watchers or history.

##### Get Operation

###### When allow_direct is false

We have extended the `io.nats.jetstream.api.v1.stream_msg_get_request` API to support loading the latest value for a specific
subject. Thus a read for `CONFIGURATION.username` becomes a `io.nats.jetstream.api.v1.stream_msg_get_request` with the
subject. Thus, a read for `CONFIGURATION.username` becomes a `io.nats.jetstream.api.v1.stream_msg_get_request` with the
`last_by_subj` set to `$KV.CONFIGURATION.auth.username`.

###### When allow_direct is true

We have introduced a new direct API that allows retrieving the last message for a subject via `$JS.APIDIRECT.GET.
<STREAM>.<SUBJECT>`. This should be used for performing all gets on a bucket if direct is enabled.

##### History

These operations require access to all values for a key, to achieve this we create an ephemeral consumer on filtered by the subject
Expand Down Expand Up @@ -360,15 +374,83 @@ first time any message has a `Pending==0`. This signal must also be sent if no d
Whatchers should support at least the following options. Languages can choose to support more models if they wish, as long as that
is clearly indicated as a language specific extension. Names should be language idiomatic but close to these below.

|Name|Description|
|----|-----------|
|`IncludeHistory`|Send all available history rather than just the latest entries|
|`IgnoreDeletes`|Only sends `PUT` operation entries|
|`MetaOnly`|Does not send any values, only metadata about those values|
|`UpdatesOnly`|Only sends new updates made, no current or historical values are sent. The End Of Initial Data marker is sent as soon as the watch starts.|
| Name | Description |
|------------------|--------------------------------------------------------------------------------------------------------------------------------------------|
| `IncludeHistory` | Send all available history rather than just the latest entries |
| `IgnoreDeletes` | Only sends `PUT` operation entries |
| `MetaOnly` | Does not send any values, only metadata about those values |
| `UpdatesOnly` | Only sends new updates made, no current or historical values are sent. The End Of Initial Data marker is sent as soon as the watch starts. |

The default behavior with no options set is to send all the `last_per_subject` values, including delete/purge operations.

#### Replicas for NATS Server 2.10 and newer

Since NATS Server 2.10 we support transforming messages as a stream configuration item. This allows us to source one
bucket from another and rewrite the keys in the new bucket to have the correct name.

To copy the keys `NEW.>` from bucket `ORDERS` into `NEW_ORDERS` we create the new stream with the following
partial config:

```json
"sources": [
{
"name": "KV_JACO",
"filter_subject": "$KV.ORDERS.NEW.>",
"subject_transforms": [
{
"src": "$KV.ORDERS.>",
"dest": "$KV.NEW_ORDERS.>"
}
]
}
],
```

This results in all messages from `ORDERS` keys `NEW.>` to be copied into `NEW_ORDERS` and the subjects rewritten on
write to the new bucket so that a unmodified KV client on `NEW_ORDERS` would just work.

As this is a `Source` and not a `Mirror` this new bucket can accept writes, we should prioritise a read-only capability
to safeguard against accidental data problems.

#### Read replica mirrors

Regional read replicas can be built using the standard mirror feature by setting `mirror_direct` to true as long as the
origin bucket also has `allow_direct`.

When a direct read is done the response will be from the rtt-nearest mirror. With a mirror added the `nats` command
can be used to verify that a alternative location is set:

```
$ nats s info KV_ORDERS
...
State:
Alternates: KV_ORDERS_NYC: Cluster: nyc Domain: hub
KV_ORDERS: Cluster: lon Domain: hub
```

Here we see a RTT-sorted list of alternatives, the `KV_ORDERS_MIRROR` is nearest to me in the RTT sorted list.

When doing a direct get the headers will confirm the mirror served the request:

```
$ nats req '$JS.API.DIRECT.GET.KV_ORDERS.$KV.ORDERS.NEW.123' ''
13:26:06 Sending request on "JS.API.DIRECT.GET.KV_ORDERS.$KV.ORDERS.NEW.123"
13:26:06 Received with rtt 1.319085ms
13:26:06 Nats-Stream: KV_ORDERS_NYC
13:26:06 Nats-Subject: $KV.ORDERS.NEW.123
13:26:06 Nats-Sequence: 12
13:26:06 Nats-Time-Stamp: 2023-10-16T12:54:19.409051084Z
{......}
```

As mirrors support subject filters these regional replicas can hold region specific keys.

As this is a `Mirror` this stream does not listen on a subject and so the only way to get data into it is via the origin
bucket.

#### API Design notes

The API here represents a minimum, languages can add local flavour to the API - for example one can add `PutUint64()` and `GetUint64()`
Expand Down

0 comments on commit 857bbd0

Please sign in to comment.