From 5e144bc8fbd49b9ce6f86ef5b22f1ee1d0ae42bb Mon Sep 17 00:00:00 2001 From: Felipe Cardozo Date: Sun, 2 Jun 2024 22:16:44 -0300 Subject: [PATCH] feat: update mirroring contents --- docs/tutorials/iot-mirroring-cloud.mdx | 73 ++++++++++--------- docs/tutorials/iot-mirroring-local.mdx | 98 +++++++++++++++----------- 2 files changed, 97 insertions(+), 74 deletions(-) diff --git a/docs/tutorials/iot-mirroring-cloud.mdx b/docs/tutorials/iot-mirroring-cloud.mdx index 88c2a53f..829d9dfd 100644 --- a/docs/tutorials/iot-mirroring-cloud.mdx +++ b/docs/tutorials/iot-mirroring-cloud.mdx @@ -63,7 +63,7 @@ Download and install mirroring binary. Use `curl` to download and install: ```bash -curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash +curl -fsS https://hub.infinyon.cloud/install/install.sh | bash ``` Make sure to add `.fluvio/bin` to the `$PATH`as specified in the installation script. @@ -87,10 +87,10 @@ Let's provision a new cluster in AWS `eu-central` using the experimental fluvio ```bash -fluvio cloud cluster create --region aws-eu-central-1 --version 0.10.15-dev-2+mirroring-b4f07fc +fluvio cloud cluster create ``` -Check cluster status: +Check the result with: ```bash @@ -99,6 +99,14 @@ fluvio cluster status Next, we'll configure the cluster to receive traffic from the edge clusters. +### Register Edge cluster + +Let's register the edge cluster `edge1` to inform our home cluster from the cloud to accept connection requests from the remote device: + + +```bash +fluvio remote register edge1 +``` ### Create the mirror topic @@ -109,7 +117,7 @@ Create a partition assignment file with an array of edge mirros we expect to con ```bash echo '[ - "edge1", "edge2" + "edge1" ]' > assignment_file.json ``` @@ -117,7 +125,7 @@ Apply the configuration file to create the topic: ```bash -fluvio topic create edge-topic --mirror-assignment assignment_file.json +fluvio topic create edge-topic --mirror-apply assignment_file.json ``` List partitions to check the assignment: @@ -131,34 +139,21 @@ It should display all partitions: ```bash TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS - edge-topic 0 5001 edge1 [] Online 0 B 0 0 0 0 [] - edge-topic 1 5001 edge2 [] Online 0 B 0 0 0 0 [] -``` - -We created 2 partitions, but we'll only use one in this tutorial. - - -### Register Edge cluster - -Let's register the edge cluster `edge1` to inform our Cloud cluster to accept connection requests from the remote device: - - -```bash -fluvio cluster remote-cluster register --type mirror-edge edge1 + edge-topic 0 5001 edge1 [] Online 0 B 0 0 0 0 [] ``` List remote clusters to check their status: ```bash -fluvio cluster remote-cluster list +fluvio remote list ``` It should show the following: ```bash - RemoteCluster RemoteType Paired Status Last Seen - edge1 mirror-edge - - - + RemoteCluster Status Last Seen + edge1 Waiting - ``` @@ -174,14 +169,14 @@ mkdir -p ~/local/projects/mirror; cd ~/local/projects/mirror ### Generate metadata for Edge Cluster -Each edge cluster requires a unique metadata file that informs the edge cluster how to connect with the target cluster. Create the config file by running the following command: +Each edge cluster requires a unique metadata file that informs the edge cluster how to connect with the home cluster. Create the config file by running the following command: ```bash -fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge1 --file edge1.json +fluvio cloud remote export edge1 --file edge1.json ``` -The Cloud cluster configuration is now complete. Next, we'll create an edge cluster and configure a mirror topic that synchronizes data to the Cloud. +The Cloud cluster configuration is now complete. Next, we'll create an edge cluster that synchronizes data to the Cloud. ## Install Edge Cluster on Local VM @@ -214,7 +209,7 @@ Download binaries: ```bash -curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash +curl -fsS https://hub.infinyon.cloud/install/install.sh | bash ``` Add to path: @@ -235,11 +230,18 @@ fluvio version ### Start Edge Cluster -We'll use the metadata `edge1` to start the edge cluster: +First we will start the cluster: + + +```bash +fluvio cluster start +``` + +Then, we'll connect to the Cloud with the metadata `edge1`: ```bash -fluvio cluster start --read-only edge1.json +fluvio home connect --file edge1.json ``` Let's check the partitions: @@ -253,7 +255,7 @@ The edge device should show the following partition: ```bash TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS - edge-topic 0 5001 upstream:0 [] Online 0 B 0 0 0 0 [] + edge-topic 0 5001 0:public_endpoint [] Online 0 B 0 0 0 0 [] ``` --- @@ -361,12 +363,19 @@ Restart the cluster: ```bash -fluvio cluster upgrade --read-only edge1.json +fluvio cluster upgrade ``` ### Consume from edge cluster -On the `edge` terminal, consume from the cluster: +First, on the `edge` terminal, check the status of the home cluster from the Cloud: + +```bash +HOME ROUTE STATUS LAST SEEN +home localhost:30003 Online 3s +``` + +Then, consume from the edge cluster: ```bash @@ -394,4 +403,4 @@ Join us on [Discord] if you have questions, or would like to suggest new improve * ["IoT Mirroring - Raspberry Pi to a Local Cluster"] -["IoT Mirroring - Raspberry Pi to a Local Cluster"]: /tutorials/iot-mirroring-local +["IoT Mirroring - Raspberry Pi to a Local Cluster"]: /docs/tutorials/iot-mirroring-local.mdx diff --git a/docs/tutorials/iot-mirroring-local.mdx b/docs/tutorials/iot-mirroring-local.mdx index 11fab786..ffb59896 100644 --- a/docs/tutorials/iot-mirroring-local.mdx +++ b/docs/tutorials/iot-mirroring-local.mdx @@ -4,18 +4,18 @@ title: "IoT Mirroring - Local" description: "Reliable IoT monitoring from movable or static Edges sensors." --- -This advanced tutorial reuqires a Raspberry Pi and a local installation of your collector cluster running on Kubernetes. +This advanced tutorial requires a Raspberry Pi and a local installation of your collector cluster running on Kubernetes. Checkout the basic version at: ["IoT Mirroring - Cloud"][iot-mirroring-cloud]. ## Raspberry Pi to Local Cluster -This section will use Raspberry Pi v3 running Ubuntu 32-bit as the edge device and our local machine for the target cluster. -Let's start with installing and configuring the target cluster. +This section will use Raspberry Pi v3 running Ubuntu 32-bit as the edge device and our local machine for the home cluster. +Let's start with installing and configuring the home cluster. -## Install Target Cluster on Local Machine +## Install Home Cluster on Local Machine -Installing the target cluster on Linux or Mac requires Kubernetes. Use the following instructions to set up Kubernetes on your local machine. +Installing the home cluster on Linux or Mac requires Kubernetes. Use the following instructions to set up Kubernetes on your local machine. * [Install Rancher Desktop for Mac](https://fluvio.io/docs/get-started/mac/#install-rancher-desktop) * [Install k3d, kubectl and helm for Linux](https://fluvio.io/docs/get-started/linux/#installing-kubernetes-cluster) @@ -47,13 +47,13 @@ Use `curl` to download and install: ```bash -curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash +curl -fsS https://hub.infinyon.cloud/install/install.sh | bash ``` Make sure to add `.fluvio/bin` to the `$PATH`as specified in the installation script. -### Start target cluster +### Start home cluster Use the fluvio binary to start the cluster: @@ -69,6 +69,23 @@ Check the result with: fluvio cluster status ``` +### Register Edge clusters + +Use the `remote-cluster` CLI to register the edge clusters (edge1 and edge2) with the upstream cluster: + +**Edge 1**: + + +```bash +fluvio cluster remote-cluster register edge1 +``` + +**Edge 2**: + + +```bash +fluvio cluster remote-cluster register edge2 +``` ### Create the mirror topic @@ -87,7 +104,7 @@ Apply the configuration file to create the topic: ```bash -fluvio topic create edge-topic --mirror-assignment assignment_file.json +fluvio topic create edge-topic --mirror-apply assignment_file.json ``` List partitions to check the assignment: @@ -105,37 +122,19 @@ It should display all partitions: edge-topic 1 5001 edge2 [] Online 0 B 0 0 0 0 [] ``` -### Register Edge clusters - -Use the `remote-cluster` CLI to register the edge clusters (edge1 and edge2) with the upstream cluster: - -**Edge 1**: - - -```bash -fluvio cluster remote-cluster register --type mirror-edge edge1 -``` - -**Edge 2**: - - -```bash -fluvio cluster remote-cluster register --type mirror-edge edge2 -``` - List remote clusters to check their status: ```bash -fluvio cluster remote-cluster list +fluvio remote list ``` It should show the following: ```bash - RemoteCluster RemoteType Paired Status Last Seen - edge1 mirror-edge - - - - edge2 mirror-edge - - - + RemoteCluster Status Last Seen + edge1 Waiting - + edge2 Waiting - ``` ### Generate Metadata for Edge Clusters @@ -146,16 +145,16 @@ Generate a metadata file for each cluster: **Edge 1**: -The target edge device is a Virtual Machine emulating an IoT device: +The home edge device is a Virtual Machine emulating an IoT device: ```bash -fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge1 --upstream host.orb.internal --file edge1.json +fluvio remote export edge1 --public-endpoint host.orb.internal --file edge1.json ``` **Edge 2**: -The target edge device is a Raspberry Pi device. You may skip this if you don't have such a device. +The home edge device is a Raspberry Pi device. You may skip this if you don't have such a device. The IP address of our machine where the upstream server is running is `192.168.79.252`. Please identify your own IP address and replace it in the command below. @@ -163,7 +162,7 @@ The target edge device is a Raspberry Pi device. You may skip this if you don't ```bash -fluvio cluster remote-cluster metadata export --topic edge-topic --mirror edge2 --upstream 192.168.79.252 --file edge2.json +fluvio remote export edge1 --public-endpoint 192.168.79.252 --file edge2.json ``` We'll transfer these files to edge devices in the following sections. @@ -203,18 +202,25 @@ On the raspberry pi, run the following command: ```bash -curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION='0.10.15-dev-2+mirroring-9961bdb' bash +curl -fsS https://hub.infinyon.cloud/install/install.sh | bash ``` Run `fluvio version` to double check. ### Start cluster -We'll use the metadata file to start the edge cluster on the Raspberry Pi: +First we will start the cluster: ```bash -fluvio cluster start --read-only edge2.json +fluvio cluster start +``` + + +Then, we'll use the metadata file on the Raspberry Pi to connect: + +```bash +fluvio home connect --file edge2.json ``` Let's check the partitions: @@ -227,8 +233,8 @@ fluvio partition list The edge device should show the following partition:: ```bash - TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS - edge-topic 0 5001 Source:upstream:5001 [] Online 0 B 11 11 11 0 [] + TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS + edge-topic 0 5001 home_name:0:public_endpoint [] Online 0 B 11 11 11 0 [] ``` @@ -357,12 +363,20 @@ Restart the cluster: ```bash -fluvio cluster upgrade --read-only edge2.json +fluvio cluster upgrade ``` ### Consume from edge cluster -On the `pi` terminal, consume from the cluster: +First, on the `pi` terminal, check the status of the target cluster: + +```bash +HOME ROUTE STATUS LAST SEEN +home localhost:30003 Online 3s +``` + + +Then, consume from the edge cluster: ```bash @@ -383,4 +397,4 @@ Produce records and observe that the mirror will resume the synchronization. [Discord]: https://discord.com/invite/bBG2dTz -[iot-mirroring-cloud]: /tutorials/iot-mirroring-cloud +[iot-mirroring-cloud]: /docs/tutorials/iot-mirroring-cloud.mdx