feat: operations

docs/concepts/batching.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 6
+sidebar_position: 10
 slug: /concepts/batching
 title: "Batching"
 ---

docs/concepts/data-consistency.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 7
+sidebar_position: 11
 slug: /concepts/data-consistency
 title: "Data Consistency"
 ---

docs/concepts/delivery-semantics.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 8
+sidebar_position: 12
 slug: /concepts/delivery-semantics
 title: "Delivery Semantics"
 ---

docs/concepts/offsets.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 5
+sidebar_position: 9
 slug: /concepts/offsets
 title: "Offsets"
 ---

docs/concepts/operations/data-retention.mdx

@@ -0,0 +1,195 @@
+---
+sidebar_position: 2
+slug: /concepts/operations/data-retentiton
+title: "Data Retention"
+---
+
+## Overview
+
+Topic data is automatically pruned when **any** of following criteria is true:
+1. Partition size exceeds the configured max partition size
+2. Elapsed time since the last write to segment has passed the configured retention time
+
+-> Data eviction operates on the segment level. If any above conditions are met, the entire segment gets removed. Only previous segments can be pruned. If your data resides in the active segment, it won't be evicted unless the segment turns to a historical (read-only) segment.
+
+
+## Configuring retention
+
+Retention is configured per-topic at the time of topic creation with `fluvio topic create`.
+
+```shell
+$ fluvio topic create -h
+Create a Topic with the given name
+
+fluvio-stable topic create [FLAGS] [OPTIONS] <name>
+        [...]
+        --retention-time <time>
+            Retention time (round to seconds) Ex: '1h', '2d 10s', '7 days' (default)
+
+        --segment-size <bytes>
+            Segment size (by default measured in bytes) Ex: `2048`, '2 Ki', '10 MiB', `1 GB`
+
+        --max-partition-size <bytes>
+            Max partition size (by default measured in bytes) Ex: `2048`, '2 Ki', '10 MiB', `1 GB`
+
+ARGS:
+    <name>    The name of the Topic to create
+```
+
+## Retention time
+
+Retention time duration can be provided as a free-form string. Check out the [`humantime` docs](https://docs.rs/humantime/2.1.0/humantime/fn.parse_duration.html) for the supported time suffixes.
+
+The default retention time is `7 days`
+
+### Example retention configurations
+* Delete old segments that are `6 hours` old
+
+%copy first-line%
+```bash
+$ fluvio topic create test1 --retention-time "6h"
+```
+
+%copy first-line%
+```bash
+$ fluvio topic list
+ NAME         TYPE      PARTITIONS  REPLICAS  RETENTION TIME  STATUS                   REASON
+ test1        computed      1          1            6h        resolution::provisioned
+```
+
+* Delete old segments that are a day old
+
+%copy first-line%
+```bash
+$ fluvio topic create test2 --retention-time "1d"
+```
+
+%copy first-line%
+```bash
+$ fluvio topic list
+ NAME         TYPE      PARTITIONS  REPLICAS  RETENTION TIME  STATUS                   REASON
+ test2        computed      1          1           1day       resolution::provisioned
+```
+
+* A very specific duration that is 1 day, 2 hours, 3 minutes and 4 seconds long
+
+%copy first-line%
+```bash
+$ fluvio topic create test3 --retention-time "1d 2h 3m 4s"
+```
+
+%copy first-line%
+```bash
+$ fluvio topic list
+ NAME         TYPE      PARTITIONS  REPLICAS  RETENTION TIME  STATUS                   REASON
+ test3        computed      1          1      1day 2h 3m 4s   resolution::provisioned
+```
+
+## Segment size
+
+Produced records persist on the SPU in file chunks that cannot exceed the segment size.
+
+If adding a new record to the active segment will would result in exceeding the segment size, it is saved into a new segment.
+
+Older segments are still available for consumption until they get pruned when the eviction condition is met.
+
+Segment size can be provided as a free form string. Check out the [`bytesize` docs](https://github.com/hyunsik/bytesize/)
+for the supported size suffixes.
+
+The default segment size is `1 GB`
+
+### Example retention configurations
+* 25 MB segment size w/ 7 day retention time
+
+%copy first-line%
+```bash
+$ fluvio topic create test4 --segment-size 25000000
+```
+
+* 36 GB segment size w/ 12 hr retention time
+
+%copy first-line%
+```bash
+$ fluvio topic create test5 --segment-size "36 GB" --retention-time 12h
+```
+
+## Max partition size
+
+Fluvio keeps tracking the disk size that a partition occupies on **SPU** node. It includes the payload and all bookkeeping data.
+If partition size exceeds the max partition size property Fluvio triggers segments eviction. The oldest segment is deleted first.
+The size enforcing operation provides `best-effort` guarantee. There might be time windows when the actual partition size may
+exceed the configured max partition size. It is recommended to configure max partitions sizes to cover up to 80% of the disk size.
+If the disk is full before the retention period is triggered, the SPU stops accepting messages and the overall health of the system
+may be compromised. The max partition size is applied to every partition in a topic. If a topic has 3 partitions
+and `--max-partition-size '10 GB'` is set, Fluvio controls the topic to not exceed 30 GB disk usage for the topic.
+
+Max partition can be provided as a free-form string. Check out the [`bytesize` docs](https://github.com/hyunsik/bytesize/)
+for the supported size suffixes.
+
+The default max partition size is `100 GB`.
+The max partition size must not be less than segment size.
+
+
+### Example retention configurations
+
+* 10 GB max partition size w/ 1 GB segment size (only 10 segments are allowed at any time)
+
+%copy first-line%
+```bash
+fluvio topic create test6 --max-partition-size '10 GB' --segment-size '1 GB'
+```
+
+## Example data lifecycle
+
+For a given topic with a retention of `7 days` using `1 GB` segments
+
+* Day 0: 2.5 GB is written (total topic data: 2.5 GB)
+
+| Topic Segment # | Segment size | Days since last write |
+|-----------------|--------------|-----------------------|
+| 0               | 1 GB         | 0                     |
+| 1               | 1 GB         | 0                     |
+| 2               | 0.5 GB       | N/A                   |
+
+* Day 6: Another 2 GB is written (total topic data: 4.5 GB,)
+
+| Topic Segment # | Segment size | Days since last write |
+|-----------------|--------------|-----------------------|
+| 0               | 1 GB         | 6                     |
+| 1               | 1 GB         | 6                     |
+| 2               | 1 GB         | 0                     |
+| 3               | 1 GB         | 0                     |
+| 4               | 0.5 GB       | N/A                   |
+
+* Day 7: 2 segments from Day 0 are 7 days old. They are pruned (total topic data: 2.5 GB)
+
+| Topic Segment # | Segment size | Days since last write |
+|-----------------|--------------|-----------------------|
+| 2               | 1 GB         | 1                     |
+| 3               | 1 GB         | 1                     |
+| 4               | 0.5 GB       | N/A                   |
+
+* Day 14: 2 segments from Day 7 are 7 days old. They are pruned (total topic data: 0.5 GB)
+
+| Topic Segment # | Segment size | Days since last write |
+|-----------------|--------------|-----------------------|
+| 4               | 0.5 GB       | N/A                   |
+
+The newest segment is left alone and only begins to age once a new segment is being written to.
+
+For a given topic with max partition size is `3 GB` and `1 GB` segments
+* 2.5 GB is written (total partition data: 2.5 GB)
+
+| Topic Segment # | Segment size |
+|-----------------|--------------|
+| 0               | 1 GB         |
+| 1               | 1 GB         |
+| 2               | 0.5 GB       |
+
+* 600 MB is written. The total size becomes 3.1 GB. The first segment is pruned.
+
+| Topic Segment #     | Segment size |
+|---------------------|--------------|
+| 1                   | 1 GB         |
+| 2                   | 1 GB         |
+| 3                   | 0.1 GB       |

docs/concepts/operations/troubleshooting.mdx

@@ -0,0 +1,64 @@
+---
+sidebar_position: 4
+slug: /concepts/operations/troubleshooting
+title: "Troubleshooting"
+---
+
+## Run cluster check
+
+To diagnose abnormal behavior, a good first step is to run `fluvio cluster check`, which checks against common problems and misconfigurations.
+
+If everything is configured properly, you should see a result like this:
+
+%copy first-line%
+```bash
+$ fluvio cluster check
+Running pre-startup checks...
+     ✅ Kubernetes config is loadable
+     ✅ Supported kubernetes version is installed
+     ✅ Supported helm version is installed
+     ✅ Can create service
+     ✅ Can create customresourcedefinitions
+     ✅ Can create secret
+     ✅ Fluvio system charts are installed
+🎉 All checks passed!
+You may proceed with cluster startup
+next: run `fluvio cluster start`
+```
+
+## Logs
+
+To discover errors, you should examine logs from the following components:
+
+### SC
+%copy first-line%
+```bash
+kubectl logs -l app=fluvio-sc
+```
+### SPU
+%copy first-line%
+```bash
+kubectl logs -l app=spu
+```
+
+## Handling Bugs
+
+### Records logs and create and GitHub Issue
+
+In the event of a bug in Fluvio, we appreciate if you could save the log output to file and create a [GitHub Issue](https://github.com/infinyon/fluvio/issues/new?assignees=&labels=bug&template=bug_report.md&title=%5BBug%5D%3A).
+
+### Reach out to community
+
+[Discord](https://discord.gg/zHsWBt5Z2n)
+
+### Restart Pods
+
+To attempt to recover from the bug, you can try restarting the K8s pods.
+
+%copy%
+```bash
+kubectl delete pod -l app=fluvio-sc
+kubectl delete pod -l app=spu
+```
+
+Fluvio pods are created by either `Deployments` or `StatefulSets`. Therefore deleting them will automatically cause new pods to be started.

docs/concepts/operations/upgrade.mdx

@@ -0,0 +1,46 @@
+---
+sidebar_position: 3
+slug: /concepts/operations/upgrade
+title: "Upgrade"
+---
+
+When upgrading to a new version of Fluvio, the first thing you need to do
+is [update the Fluvio CLI] using the following command:
+
+[update the Fluvio CLI]:
+
+%copy first-line%
+```bash
+$ fluvio update
+```
+
+After updating the CLI, we can upgrade the Fluvio _cluster_ using this command:
+
+%copy first-line%
+```bash
+$ fluvio cluster upgrade
+```
+
+Internally, the `fluvio cluster upgrade` command is actually using the
+<a href="https://helm.sh/docs/helm/helm_upgrade/" target="_blank">helm</a>
+package manager to upgrade the Fluvio helm charts installed on
+your Kubernetes cluster.
+
+You may optionally specify a specific Fluvio chart version by adding
+the `--chart-version` argument to the upgrade command, such as the following:
+
+%copy first-line%
+```bash
+$ fluvio cluster upgrade --chart-version=0.8.3
+```
+
+~> Note that upgrading is not reversible, as downgrading is not supported.
+
+The possible versions you may pass to `--chart-version` correspond to the
+available helm charts published on the Fluvio chart museum. You may view
+this list of releases quickly with the command:
+
+%copy first-line%
+```bash
+$ fluvio cluster releases list
+```

docs/concepts/partitions.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 3
+sidebar_position: 7
 slug: /concepts/partitions
 title: "Partitions"
 ---

docs/concepts/produce-consume.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 1
+sidebar_position: 5
 slug: /concepts/produce-consume
 title: "Producers and Consumers"
 ---

docs/concepts/records.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 4
+sidebar_position: 8
 slug: /concepts/records
 title: "Records"
 ---

docs/concepts/topics.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 2
+sidebar_position: 6
 slug: /concepts/topics
 title: "Topics"
 ---

docs/concepts/transformations.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 9
+sidebar_position: 13
 slug: /concepts/transformations
 title: "Transformations"
 ---